Truace
Subscribe

Evaluation of six generative large language models answering pre-analytical questions in medical microbiology

Objectives: Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in pre-analytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common pre-analytical questions in the domain of medical microbiology. Next, we propose a blueprint to evaluate LLMs in medical microbiology. Methods: We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domai…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1354· cite & verify
Evaluation of six generative large language models answering pre-analytical questions in medical microbiology
"LUA, Faculty of Food Technology Food microbiology laboratory" by Turaids is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

In brief

Researchers compared six language models on common pre-analytical questions in medical microbiology, using OpenAI's GPT-3.5 and GPT-4, Meta's Llama2-7B and Llama2-13B, and two domain-enhanced versions. Outputs were checked against expert-reviewed reference answers and scored by thirty-three medical professionals.

The findings matter because diagnostic stewardship depends on correct guidance before samples are analysed, and the study shows model quality is uneven and judged differently by seniority. The abstract does not say how such differences would affect actual patient care, so real-world consequences remain unmeasured.

Main points

  1. OpenAI models were more similar to the reference answers than Meta AI models and significantly outperformed them in accuracy and completeness.
  2. A customised GPT-4 agent matched OpenAI's general-purpose models while giving significantly shorter responses (p<0.001).
  3. Medical microbiologists and experienced infectious disease specialists gave significantly lower ratings than trainees and medical students (p<0.001).
  4. The authors propose a blueprint for evaluating LLMs in medical microbiology.

The problem

Six LLMs performed inconsistently on pre-analytical microbiology questions, and experienced specialists rated responses lower than trainees did, so the answers need critical appraisal before use.

The rundown

The evaluation used a hybrid framework. It measured response length by word count, compared text similarity with reference answers using BLEU, ROUGE and BERTScore, and collected human ratings of accuracy and completeness on a 1-7 scale.

Besides the general-purpose models from OpenAI and Meta AI, the comparison included a customised GPT-4 agent and Llama2-7B Medical. The customised agent's shorter answers did not reduce its measured performance relative to the general-purpose OpenAI models.

The authors conclude that differences in perceived quality across professional groups call for critical appraisal when LLMs are consulted for pre-analytical guidance, and they offer an evaluation blueprint for future work.

Sources

  1. Peer-reviewedClinical Microbiology and Infection2026-10-13

The debate