Truace
Subscribe

Benchmarking commercial large language models for gene-disease-phenotype extraction from full-text human genetics literature

Manual curation of gene-disease-phenotype relationships from the human genetics literature is a persistent bottleneck for maintaining its bioinformatics databases. Whereas large language models (LLMs) offer a promising alternative, there is currently no systematic benchmark that evaluates whether state-of-the-art commercial LLMs can perform this task reliably on the full-text articles. To address this gap, we introduce a standardized benchmark comprising 406 full-text articles covering 180 congenital heart disea…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1862· cite & verify
Benchmarking commercial large language models for gene-disease-phenotype extraction from full-text human genetics literature
Gene A as an example of a haplosufficient gene (HS). This gene is active in brain, heart and liver by Alfonso Prado-Cabrero. CC0 · http://creativecommons.org/publicdomain/zero/1.0/deed.en

In brief

Researchers built a benchmark of 406 full-text articles covering 180 genes linked to congenital heart disease. They used it to test seven commercial language models on pulling out structured gene, disease, and phenotype information, comparing outputs against a curated database.

The result matters because curation of genetics databases is slow and manual, and the tested models performed well on this task. The abstract does not describe real-world deployment, and the evaluation covers one disease area, so how the results carry over to other fields remains unclear.

Main points

  1. The benchmark includes 406 full-text articles covering 180 congenital heart disease-associated genes.
  2. Seven models were tested: GPT-4o, Claude-Opus-4, DeepSeek-R1, Grok-4, Qwen-3.5, Gemini-2.5 (Pro), and GPT-5.
  3. Grok-4 scored highest at 97.6% overall accuracy, while the lowest-performing model reached roughly 88%.
  4. The evaluation framework uses fuzzy matching to account for synonyms and partial matches.
  5. The dataset, framework, and model outputs are released online for reproducible follow-up studies.

The gain

Commercial LLMs extracted structured gene, disease, and phenotype fields from full-text genetics papers with high accuracy, which could ease the manual curation bottleneck.

The rundown

The study responds to the absence of a systematic benchmark for whether commercial LLMs can reliably handle gene-disease-phenotype extraction from full-text articles rather than abstracts alone.

Models were asked to extract structured gene, disease, and phenotype fields, and results were scored with a multi-dimensional framework that tolerates synonyms and partial matches.

Even the weakest model, at about 88%, outperformed many earlier zero-shot or n-shot prompting benchmarks in biomedical relation extraction, according to the authors. The paper has two components: a human genetics field benchmark against a curated database and the evaluation framework itself.

Sources

  1. Peer-reviewedQuantitative Biology2026-08-17

The debate