aceReproducible but Incoherent: A Reference Standard-Free Reliability Benchmark of LLM-Assisted Nutritional Estimation Across 20 Model Configurations
Nutritional estimation pipelines built on large language models are entering biomarker research, but free-text meal descriptions have no ground truth, so outputs cannot be validated against a reference standard. What can be established without one is whether the same input returns the same number. We submitted 191 de-identified Spanish meal descriptions, byte-identically, three times each, at temperature 0, to 20 model configurations across two inference platforms (12,119 calls), and measured test–retest reliabi…
All 20 LLM configurations tested gave more repeatable energy estimates for the same meal description than the audited cascade comparator, with the best reaching a test-retest ICC of 1.0000.
High reproducibility did not ensure internal coherence: the most reproducible model departed from its own macronutrients by more than 10% in 65.58% of responses, versus 3.80% for the cascade.
The coherence measure is not a validated criterion, and the binary acceptance gate is sensitive to the chosen bound, so it cannot establish the accuracy of nutritional estimates.
Evidence
- Peer-reviewedMachine Learning and Knowledge Extraction2026-10-15
How should this claim be treated?
Truvace Impact Record TRV-2026-1669, v1: “Reproducible but Incoherent: A Reference Standard-Free Reliability Benchmark of LLM-Assisted Nutritional Estimation Across 20 Model Configurations.” Truvace, 2026-10-17. /record/TRV-2026-1669 (accessed at citation time). sha256 ad7e406144ac8410…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1669 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.