aceEntity-centric evaluation of large language model responses for medical question-answering tasks
Large language models (LLMs) are increasingly evaluated for clinical question-answering (QA) tasks, yet benchmark accuracy alone provides limited assurance of clinical reliability. Existing evaluation metrics often fail to capture whether model reasoning preserves patient-specific context and diagnostic intent, while many require external references or manual annotation, limiting scalability and real-world applicability. To address this gap, this study proposes EntQA, a reference-free, entity-centric metric that…
EntQA, a reference-free entity-centric metric, correlated positively with accuracy and model scale across five medical QA benchmarks, where overlap- and embedding-based metrics often did not, offering scalable assessment without manual annotation.
The evidence is limited to correlations on five benchmarks and seven models from a single family (Qwen 2.5 Instruct), and the correlation with model scale was modest (ρ = 0.252).
Evidence
- Peer-reviewedPLOS Digital Health2026-10-06
How should this claim be treated?
Truvace Impact Record TRV-2026-1811, v1: “Entity-centric evaluation of large language model responses for medical question-answering tasks.” Truvace, 2026-10-18. /record/TRV-2026-1811 (accessed at citation time). sha256 5b43084843af5315…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1811 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.