+ POLICY — Keir Starmer has said ministers should be able to “look every parent in the eye” and pledge that tech can cre…− POLICY — Artificial intelligence poses a “Hiroshima”-style risk to humanity if governments do not agree to curb how it…− CLIMATE — Artificial intelligence is often associated with ludicrous amounts of electricity, and therefore planet-heati…− EDUCATION — While many schools in England have banned smartphones, in Estonia – regarded as the new European education po…− EDUCATION — In a Cambridge classroom, Joseph, 10, trained his AI model to discern between drawings of apples and drawings…− EDUCATION — OpenAI CEO Sam Altman recently told a US podcast that if he was graduating today, “I would feel like the luck…+ EDUCATION — I disagree with the decision of lecturers to use artificial intelligence to create teaching materials (‘We co…− BUSINESS — Americans are growing worried about what artificial intelligence portends for their futures. Eight in 10 Amer…
TruaceTracing the truth around AISunday, July 12, 2026
Subscribe
Health·P Space·Evidence-backed problem·Published 2026-07-12

Large language models encode clinical knowledge

Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We p…

TRV-2026-0061Peer-reviewedPermanent record — cite & verify
Large language models encode clinical knowledge

"The Data Science curvilinear triangle" by Carlo.lauro is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

The quick read

Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks.

In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA.

Main points
  • Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high.
  • Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks.
  • Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA.
Problem

Large language models encode clinical knowledge: We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.

The rundown

Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.

Sources

The debate