Reliability and coherence of LLM-assisted nutritional estimation from meal descriptions

Source article: Reproducible but Incoherent: A Reference Standard-Free Reliability Benchmark of LLM-Assisted Nutritional Estimation Across 20 Model Configurations

Nutritional estimation pipelines built on large language models are entering biomarker research, but free-text meal descriptions have no ground truth, so outputs cannot be validated against a reference standard. What can be established without one is whether the same input returns the same number. We submitted 191 de-identified Spanish meal descriptions, byte-identically, three times each, at temperature 0, to 20 model configurations across two inference platforms (12,119 calls), and measured test–retest reliabi…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1669· cite & verify

Reproducible but Incoherent: A Reference Standard-Free Reliability Benchmark of LLM-Assisted Nutritional Estimation Across 20 Model Configurations — "Lunch at Le Commensal" by moria is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.

Trace impact readingContested

P 66

Both sides are scored from claims and sources, not community votes.

G 67

In brief

Researchers sent 191 de-identified Spanish meal descriptions, each three times at temperature 0, to 20 LLM configurations on two inference platforms, totalling 12,119 calls. They measured whether the same input produced the same number, how models agreed with one another, and whether reported energy matched reported macronutrients.

The work matters because LLM nutrition pipelines are entering biomarker research where no ground truth exists for free-text meals. The authors conclude that repeatability alone cannot justify use and that a reference-standard-free coherence check is also needed, though their coherence measure is only a ranked screening dimension rather than a validated criterion.

Main points

The study used 191 Spanish meal descriptions submitted three times each to 20 model configurations across two platforms.
Every model exceeded the audited cascade on energy test-retest reliability under both comparators.
The most reliable model also ranked first of 20 in incoherence at every bound from 5% to 20%.
A pass/fail coherence gate was unstable, with 3 to 17 of 20 configurations passing depending on the bound.
The ranking of models by coherence was stable, with Kendall tau of at least 0.74 between any two bounds.

The gain

All 20 LLM configurations tested gave more repeatable energy estimates for the same meal description than the audited cascade comparator, with the best reaching a test-retest ICC of 1.0000.

The problem

High reproducibility did not ensure internal coherence: the most reproducible model departed from its own macronutrients by more than 10% in 65.58% of responses, versus 3.80% for the cascade.

The rundown

The design deliberately avoided a reference standard because free-text meal descriptions have no ground truth. Instead it tested measurable properties: test-retest reliability by ICC(1,1), cross-model agreement, Atwater coherence between energy and macronutrients, and the reliability gradient across input specificity.

The central finding is a dissociation between two nearly orthogonal axes. One configuration scored an ICC of 1.0000 yet had the largest share of incoherent responses under the 4/4/9 Atwater approximation, at 65.58%, compared with 3.80% for the audited cascade.

Because a binary acceptance threshold changed outcomes sharply across bounds, the authors report coherence as a ranked screening dimension with a decision curve. They recommend at least one reference-standard-free coherence check alongside reliability before authorising an LLM-assisted nutritional pipeline.

What this doesn’t fix

The coherence measure is not a validated criterion, and the binary acceptance gate is sensitive to the chosen bound, so it cannot establish the accuracy of nutritional estimates.