Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-0061
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-07-12T20:50:42.239655Z
status: published
lens: p_space
sector: health
headline: Large language models encode clinical knowledge
dek: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model 1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM 2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accu…
gain_reading: (none)
problem_reading: Large language models encode clinical knowledge: We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.
limitation: Historical evidence reading: the cited study may be limited by its design, population, period, or setting, and later research may report different effects.
tag: Evidence-backed problem
key_points: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. | Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. | Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA.
rundown: Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks.

Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias.
sources:
- peer_reviewed | Nature | https://doi.org/10.1038/s41586-023-06291-2 | 2023-07-12
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

d2e299a11abce5f7af461d29f16e89d681ba7455ed2c0ad032e2687007be8ae9

previous

0000000000000000000000000000000000000000000000000000000000000000