Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1354
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-14T18:14:22.703789Z
status: published
lens: p_space
sector: health
headline: Evaluation of six generative large language models answering pre-analytical questions in medical microbiology
dek: Objectives: Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in pre-analytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common pre-analytical questions in the domain of medical microbiology. Next, we propose a blueprint to evaluate LLMs in medical microbiology.
Methods: We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domai…
gain_title: (none)
problem_title: Six LLMs performed inconsistently on pre-analytical microbiology questions, and experienced specialists rated responses lower than trainees did, so the answers need critical appraisal before use.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: Six LLMs performed inconsistently on pre-analytical microbiology questions, and experienced specialists rated responses lower than trainees did, so the answers need critical appraisal before use.
problem_evidence: medical microbiologists and experienced infectious disease specialists assigned significantly lower scores compared to infectious disease trainees and medical students
quick_read: Researchers compared six language models on common pre-analytical questions in medical microbiology, using OpenAI's GPT-3.5 and GPT-4, Meta's Llama2-7B and Llama2-13B, and two domain-enhanced versions. Outputs were checked against expert-reviewed reference answers and scored by thirty-three medical professionals.

The findings matter because diagnostic stewardship depends on correct guidance before samples are analysed, and the study shows model quality is uneven and judged differently by seniority. The abstract does not say how such differences would affect actual patient care, so real-world consequences remain unmeasured.
limitation: The study shows that ratings varied markedly between professional groups, so perceived quality is not uniform and a single assessment of the models' reliability may be unreliable.
tag: Evidence-backed problem
key_points: OpenAI models were more similar to the reference answers than Meta AI models and significantly outperformed them in accuracy and completeness. | A customised GPT-4 agent matched OpenAI's general-purpose models while giving significantly shorter responses (p<0.001). | Medical microbiologists and experienced infectious disease specialists gave significantly lower ratings than trainees and medical students (p<0.001). | The authors propose a blueprint for evaluating LLMs in medical microbiology.
rundown: The evaluation used a hybrid framework. It measured response length by word count, compared text similarity with reference answers using BLEU, ROUGE and BERTScore, and collected human ratings of accuracy and completeness on a 1-7 scale.

Besides the general-purpose models from OpenAI and Meta AI, the comparison included a customised GPT-4 agent and Llama2-7B Medical. The customised agent's shorter answers did not reduce its measured performance relative to the general-purpose OpenAI models.

The authors conclude that differences in perceived quality across professional groups call for critical appraisal when LLMs are consulted for pre-analytical guidance, and they offer an evaluation blueprint for future work.
sources:
- peer_reviewed | Clinical Microbiology and Infection | https://doi.org/10.1016/j.cmi.2026.09.035 | 2026-10-13
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

72ec82b6e45e864e46d02913a7271f634ca4c3e4c135cd93f0fb2297993416a4

previous

0000000000000000000000000000000000000000000000000000000000000000