Truace
Subscribe

Genolator beat GPT 4.1 on 34,743 protein yes-or-no questions, a benchmark test found

Study title: Genolator enables protein function interpretation using a multimodal large language model fusing genomic and structural interpretation with natural language interaction

Researchers built Genolator, a multimodal model based on Llama that combines DNA embeddings from Evo2, amino acid embeddings from ESM2 or structure-aware embeddings from PST, and an English question. It was fine-tuned on more than 365,000 question-answer pairs built from Gene Ontology terms covering subcellular localization, molecular function and biological processes. On 34,743 held-out questions about unseen proteins, the best version scored an F1 of about 0.98 and an MCC of 0.959.

Read the study abstract

Background Decoding the genetic code to unveil its genome functionality is a monumental task which would greatly advance the understanding of disease mechanisms and development of targeted treatments. Although large language models (LLMs) have transformed natural language processing across diverse domains, translating the complex language of DNA into human-readable form remains challenging due to genomic data complexity and unexplored regions of the human genome. Current language models either are capable of pro…

Truvace record3 min read1 sourceEvidence: Moderate · 57/100 — Benchmark or lab evaluation · n = 365,000 · 1 independent sourcePeer-reviewedTRV-2026-1133 · cite or check it
Genolator enables protein function interpretation using a multimodal large language model fusing genomic and structural interpretation with natural language interaction
Breakage-Reunion-Domain-of-Streptococcus-pneumoniae-Topoisomerase-IV-Crystal-Structure-of-a-Gram-pone.0000301.s003 by Laponogov I, Veselkov D, Sohi M, Pan X, Achari A, Yang C, Ferrara J, Fisher L, Sanderson M. CC BY 2.5 · https://creativecommons.org/licenses/by/2.5

In brief

General-purpose models did far worse. GPT 4.1 reached an MCC of only 0.179. In a case study it misidentified BRCA1 as TP53 and HTT and IHH as COL1A1 and MPZ, and it claimed to have run BLAST searches whose results were fabricated or showed no significant similarity. A tuned XGBoost classifier on PST and BioBERT embeddings reached an MCC of 0.913, so Genolator's edge over the specialist baseline was modest.

Open-ended answers were less strong. Predicted terms overlapped the ground truth for 78.51% of biological process questions, 89.19% for cellular component and 89.59% for molecular function, but exact matches were only 2.51% to 5.83%. The authors present the tool as a way to make genomic embeddings accessible through natural language for research on uncharacterized proteins, not as a validated clinical tool.

Main points

  1. Genolator was fine-tuned on over 365,000 question-answer pairs, including both genuine associations and deliberately constructed denial questions.
  2. The holdout test set had 34,743 questions, and a KMeans-based split placed test proteins in distinct regions of the multimodal embedding space to mimic proteins of unknown function.
  3. The Evo2 + ESM2 variant reached an F1 of 0.97 and an MCC of 0.948; the projector-only ablations reached MCCs of 0.895 (ESM2) and 0.924 (PST).
  4. GPT 4.1 identified the INS (insulin) sequence correctly but misidentified the longer BRCA1 and IHH sequences and described BRCA1 as involved in transport across membranes.
  5. Untrained Llama-3 8B could not be benchmarked because long DNA inputs exceeded memory and time limits even on an NVIDIA H100, and its answers included amino-acid gibberish and irrelevant follow-up questions.
  6. Exact matches on open-ended questions were only 5.44% for biological process, 5.83% for cellular component and 2.51% for molecular function.
  7. Biological process was the hardest aspect; for example, regulation of DNA-templated transcription was generated 97 times against 444 in the ground truth.
  8. Zeroing the PST tokens raised test loss by 3.31% in the Evo2 + PST model, versus 0.56% when Evo2 tokens were removed.

The gain

On 34,743 held-out questions about unseen proteins, Genolator scored an MCC of 0.959 on confirm-or-deny function questions, versus 0.913 for a specialist XGBoost classifier and 0.179 for GPT 4.1.

The full story

Genolator is a finetuned Llama model that receives an English question plus embeddings of a gene's DNA sequence (Evo2), its amino acid sequence (ESM2) and, in the best variant, a structure-infused protein embedding (PST). Token projectors fuse these inputs, and the model replies in English. Answers were derived from three Gene Ontology aspects, which a GPT 4.1 model rephrased into plain language for training.

In the confirm-or-deny task the best model reached an MCC of 0.959 and an F1 of about 0.98, ahead of the XGBoost classifier at 0.913 and GPT 4.1 at 0.179. Across categories, molecular function was easiest and biological process hardest. The projector-only version outperformed GPT 4.1 but was only slightly different from XGBoost.

In the GPT 4.1 case study, the insulin gene was handled correctly, but for BRCA1 and IHH the model reported identifications with no significant sequence similarity to the reference. The authors note that GPT 4.1 likely saw many genomic sequences in training yet still could not interpret them in context.

Hidden-state analysis showed confirmed and denied associations separating clearly, and related terms clustering together, such as nuclear localizations and enzyme activities. Attention analysis showed Evo2 tokens receiving mean attention above 0.6 in most layers from 2 to 31 in the ESM2 model, and the token ablation showed the structure-aware PST tokens mattered more than Evo2 tokens in the PST model.

The authors say open-ended evaluation relied on GPT 4.1 as a judge to map answers to GO Slim terms because expert review of thousands of predictions was not feasible. Hyperparameters of the token projectors and Llama model were not tuned, and further work is needed for multiple species.

Sources

  1. Peer-reviewedGenome Biology2026-09-16

The debate