Truace
Subscribe

Semantic Clinical Knowledge Augmentation Improves Medical Question Answering Across Commercial and Open Large Language Models

Objective To evaluate whether semantic clinical knowledge augmentation through Semantic Clinical Artificial Intelligence (SCAI) improves medical question answering across commercial and open large language models, and to examine how a reasoning distilled model performs on the same benchmark. Materials and methods We evaluated Google Gemini, Microsoft Copilot, Meta Llama 3 70B, and DeepSeek-R1-Distill-Llama-70B on official text only United States Medical Licensing Exam (USMLE) sample questions from Step 1 (n = 87…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1676· cite & verify
Semantic Clinical Knowledge Augmentation Improves Medical Question Answering Across Commercial and Open Large Language Models
Hospital Universitari Doctor Peset, València 10 by 19Tarrestnom65. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0

In brief

Researchers tested four language models, Google Gemini, Microsoft Copilot, Meta Llama 3 70B, and DeepSeek-R1-Distill-Llama-70B, on official text-only USMLE sample questions from Steps 1, 2 CK, and 3. Each model was run with and without SCAI, a semantic knowledge system that supplies clinical context to the host model. Three of the four models improved significantly, while Copilot barely changed.

The result suggests that grounding general models in structured medical knowledge can reduce wrong answers, which the authors treat as confabulations. It remains unclear how this benchmark performance carries over to real clinical care, since the test covers only multiple-choice sample questions, and the authors caution that strong general reasoning does not automatically mean healthcare readiness.

Main points

  1. The test set was 313 text-only items: 87 from Step 1, 103 from Step 2 CK, and 123 from Step 3.
  2. SCAI combines HD-NLP parsing, graph and knowledge graph embeddings, a trained SCAI LLM reasoner, and a semantic knowledge base of 33,023,902 triples across 246 relation labels.
  3. Copilot moved only from 93.6% to 94.2%, and its gain was not statistically significant.
  4. The best single result was Llama 3 70B plus SCAI on Step 3, at 121 of 123 correct (98.4%).
  5. Gemini plus SCAI was the numerically strongest commercial system.

The gain

Pairing language models with the SCAI semantic knowledge pipeline raised accuracy on 313 USMLE sample questions for Gemini, Llama 3 70B, and DeepSeek-R1-Distill, with statistically significant gains on all three exam steps.

The rundown

The study compared the models with and without SCAI on the same items. Responses were scored against the answer key with secondary human verification, and because every model answered every text-only item, each incorrect response was counted as a confabulation.

Gains were largest for the weakest baseline. DeepSeek-R1-Distill rose from 55.3% to 81.8%, Llama 3 70B from 82.4% to 92.0%, and Gemini from 90.7% to 97.1%. Within-model gains were significant on all three exam steps for these three models.

The authors conclude that a dedicated SCAI LLM generating clinical context for a host LLM is a viable pipeline, and that general reasoning strength alone does not establish readiness for healthcare use.

Sources

  1. Peer-reviewedIEEE Journal of Biomedical and Health Informatics2026-10-09

The debate