Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-0061
version: 4
kind: sources_changed
reason: Source set updated
timestamp: 2026-07-12T20:50:57.306853Z
status: published
lens: p_space
sector: health
headline: Large language models encode clinical knowledge
dek: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accu…
gain_reading: (none)
problem_reading: Large language models encode clinical knowledge: We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.
limitation: Historical evidence reading: the cited study may be limited by its design, population, period, or setting, and later research may report different effects.
tag: Evidence-backed problem
key_points: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. | Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. | Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA.
rundown: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks.

Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.
sources:
- peer_reviewed | Diagnostics | https://doi.org/10.3390/diagnostics14141468 | 2024-07-09
- peer_reviewed | Journal of Artificial Intelligence, Applications, and Innovations | https://doi.org/10.61838/jaiai.1.4.5 | 2024-01-01
- peer_reviewed | Nature | https://doi.org/10.1038/s41586-023-06291-2 | 2023-07-12
- peer_reviewed | Nature Medicine | https://doi.org/10.1038/s41591-024-03423-7 | 2025-01-08
prev: 61798451a51e74eb2a5965f5125f40351d2aff4ae7b9755d2f83699485f4ff0a

sha256

86ac56b4b9a1750b958bed8a7fa6e5dae0a71d78b6d62f8697d5ed3b356e1d2d

previous

61798451a51e74eb2a5965f5125f40351d2aff4ae7b9755d2f83699485f4ff0a