LLM accuracy on harm reduction advice for injecting oral medications

Source article: Evaluating Harm Reduction Advice From Large Language Models

Introduction: Large language models (LLM) and generative Artificial Intelligence are increasingly providing health information. We have applied a novel method to evaluate the harm reduction information generated by popular LLMs. We focused on the filtration of oral medications for injection and the availability of local harm reduction services. Methods: We designed a scoring rubric to evaluate the accuracy of the information provided by each LLM and applied it to four unique, commonly used LLMs. Results: Gemini…

By the Truvace deskOctober 16, 20262 min read1 sourcePeer-reviewedTRV-2026-1608· cite & verify

Evaluating Harm Reduction Advice From Large Language Models — "Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

Trace impact readingContested

P 71

Both sides are scored from claims and sources, not community votes.

G 69

In brief

Researchers built a scoring rubric to test how accurately four widely used large language models answered harm reduction questions. The questions covered filtering oral medications before injection and finding local harm reduction services. Gemini 2.5 Flash scored 87% and ChatGPT-4o 77%, while DeepSeek R1 Distill Llama 3.3 70B scored 22% and Llama 4 scored 18%.

The results matter because people increasingly turn to LLMs for health information, and quality varied sharply between models. Some answers were accurate and non-judgmental, while others contained misinformation or no answer at all. The authors conclude that in-person discussion with harm reduction specialists is still needed.

Main points

Four commonly used LLMs were scored with a purpose-built rubric on harm reduction accuracy.
Gemini 2.5 Flash scored 87% and ChatGPT-4o 77%, versus 22% for DeepSeek R1 Distill and 18% for Llama 4.
Gemini and ChatGPT-4o gave accurate, location-specific information on local harm reduction services.
DeepSeek R1 and Llama 4 gave limited to no responses on general and situational questions.
The rubric can be adapted to other harm reduction topics for ongoing evaluation.

The gain

Gemini 2.5 Flash and ChatGPT-4o gave comprehensive, accurate micron filter instructions and correct local harm reduction service information, in a non-judgmental manner.

The problem

DeepSeek R1 Distill and Llama 4 gave little or no filter instructions, some models generated misinformation, and some refused questions because of built-in safety parameters.

The rundown

The study focused on two areas: filtering oral medications for injection with micron filters, and the availability of local harm reduction services. It also included general information and situational questions.

The higher-scoring models, Gemini 2.5 Flash and ChatGPT-4o, gave comprehensive filter instructions and accurate local service details. DeepSeek R1 and Llama 4 did not provide instructions on filter use, and gave limited or no responses to general and situational questions.

Across the models, most answers were reasonable and delivered in a non-judgmental manner. Some models nonetheless produced misinformation, and some declined to answer because of built-in safety parameters. The authors present the rubric as a method that can be reused for ongoing evaluation of other harm reduction topics.

What this doesn’t fix

The study tested only four LLMs on a limited set of harm reduction topics, and the models also produced misinformation, so the findings do not show that any model is a safe replacement for specialist advice.