Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-0061
version: 8
kind: retracted
reason: Model backfill: source did not support a publishable AI-impact claim
timestamp: 2026-07-13T05:17:34.118928Z
status: archived
lens: trace
sector: health
headline: Large language models encode clinical knowledge
dek: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We p…
gain_title: Large language models including PaLM 540-billion parameter model and Flan-PaLM encode clinical knowledge and can answer medical questions spanning professional medicine, research and consumer queries as measured on MultiMedQA.
problem_title: LLM answers to medical questions risk failures in factuality, comprehension, reasoning, and introduce possible harm and bias that are not captured by automated evaluations based on limited benchmarks.
trace_subject: LLM performance and safety when answering medical questions in clinical and consumer contexts
gain_reading: Large language models including PaLM 540-billion parameter model and Flan-PaLM encode clinical knowledge and can answer medical questions spanning professional medicine, research and consumer queries as measured on MultiMedQA.
gain_evidence: (none)
problem_reading: LLM answers to medical questions risk failures in factuality, comprehension, reasoning, and introduce possible harm and bias that are not captured by automated evaluations based on limited benchmarks.
problem_evidence: (none)
quick_read: Researchers introduced MultiMedQA, a benchmark that combines six existing medical question answering datasets covering professional medicine, research and consumer queries with a new set of online-searched questions called HealthSearchQA. They evaluated Pathways Language Model 1, a 540-billion parameter LLM, and its instruction-tuned variant Flan-PaLM on this collection to move beyond limited automated tests.

Clinical use sets a high bar because incorrect medical answers can cause harm and reflect bias, so automated scores alone are insufficient. The proposed human evaluation framework along factuality, comprehension, reasoning, harm and bias matters for patient safety, but the supplied text does not report how often models fail, which populations are most affected, or whether the framework reduces risk in practice.
limitation: Article text provided is limited to abstract-level description and does not report quantitative performance, error rates, or mitigation results for harm and bias.
tag: Model-validated trace
key_points: MultiMedQA combines six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset HealthSearchQA of medical questions searched online. | Study evaluates Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant Flan-PaLM on MultiMedQA. | Authors propose human evaluation framework for model answers along axes including factuality, comprehension, reasoning, possible harm and bias to address limits of automated evaluations.
rundown: Researchers introduced MultiMedQA, a benchmark that combines six existing medical question answering datasets covering professional medicine, research and consumer queries with a new set of online-searched questions called HealthSearchQA. They evaluated Pathways Language Model 1, a 540-billion parameter LLM, and its instruction-tuned variant Flan-PaLM on this collection to move beyond limited automated tests.

Clinical use sets a high bar because incorrect medical answers can cause harm and reflect bias, so automated scores alone are insufficient. The proposed human evaluation framework along factuality, comprehension, reasoning, harm and bias matters for patient safety, but the supplied text does not report how often models fail, which populations are most affected, or whether the framework reduces risk in practice.
sources:
- peer_reviewed | BioMedInformatics | https://doi.org/10.3390/biomedinformatics6020013 | 2026-03-13
- peer_reviewed | Diagnostics | https://doi.org/10.3390/diagnostics14141468 | 2024-07-09
- peer_reviewed | Journal of Artificial Intelligence, Applications, and Innovations | https://doi.org/10.61838/jaiai.1.4.5 | 2024-01-01
- peer_reviewed | Nature | https://doi.org/10.1038/s41586-023-06291-2 | 2023-07-12
- peer_reviewed | Nature Medicine | https://doi.org/10.1038/s41591-024-03423-7 | 2025-01-08
prev: c94bf0c05062c36f59797f7cb2a9835394ac18f22e212b5205bf67d24a44cd01

sha256

fe5a733a1c19feb505f6a1acf022a2bacd2b6d4a7d62e18bd562539386a2261b

previous

c94bf0c05062c36f59797f7cb2a9835394ac18f22e212b5205bf67d24a44cd01