52/100

Moderate evidence

Cite
More
EmailLinkedInX

Only one of five AI chatbots ran a full NMR oil analysis on its own, a study finds

Study title: LLMs in the Laboratory: Benchmarking Conversational AI for Multivariate Analysis of NMR Spectroscopic Data

Evidence type
Benchmark or lab evaluation
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: LLMs in the Laboratory: Benchmarking Conversational AI for Multivariate Analysis of NMR Spectroscopic Data

A study in Analytical Chemistry compared five large language models on a complete multivariate analysis of low-field NMR data from edible vegetable oils. The models were Claude Opus 4.6, Gemini 3.1 Pro, ChatGPT (GPT-5.4 Thinking), Grok-4 and Qwen 3.5, and each was prompted in natural language to explore data, classify samples, validate models and interpret results.

Read the study abstract

This study presents a comparative evaluation of five state-of-the-art large language models (LLMs) (Claude Opus 4.6, Gemini 3.1 Pro, Grok-4, ChatGPT (GPT-5.4 Thinking), and Qwen 3.5) for their ability to perform key chemometric tasks through natural language interaction. Using a low-field nuclear magnetic resonance (LF-NMR) dataset of edible vegetable oils, the models were prompted to execute a complete multivariate analysis workflow, including data exploration, unsupervised and supervised classification, model…

LLMs in the Laboratory: Benchmarking Conversational AI for Multivariate Analysis of NMR Spectroscopic Data
"Particle accelerator DSC09089" by David Monniaux, CC BY-SA 3.0.

49Gain side

How the two sides scoreProblem scores higher

54Problem side

The gain

In a benchmark of five chatbots on low-field NMR data from edible vegetable oils, Claude Opus 4.6 ran the whole analysis pipeline on its own and correctly predicted all nine unlabeled samples.

The problem

In a benchmark of five chatbots on low-field NMR data from edible vegetable oils, the open-source Qwen 3.5 failed unrecoverably at the supervised classification stage and could not finish the workflow, and Grok-4 showed limits in data handling autonomy.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Claude Opus 4.6 was the only model to run the whole pipeline autonomously, and it correctly predicted all nine unlabeled samples. Gemini 3.1 Pro and ChatGPT reached 100% or near-100% classification accuracy, though within fallback or native workflows. Qwen 3.5 failed at the supervised classification stage and could not complete the work.

The authors conclude that fluent language is not a reliable sign of analytical competence, and that general-purpose benchmarks may miss weaknesses in domain-specific chemometric work. They say critical validation and human oversight remain essential when these tools are used in spectroscopy workflows.

Main points

  1. Five models were tested: Claude Opus 4.6, Gemini 3.1 Pro, Grok-4, ChatGPT (GPT-5.4 Thinking) and Qwen 3.5.
  2. The task was a full multivariate workflow on a low-field NMR dataset of edible vegetable oils, from data exploration to interpretation.
  3. Claude Opus 4.6 alone completed the whole pipeline autonomously and predicted all nine unlabeled samples correctly.
  4. Gemini 3.1 Pro and ChatGPT reached 100% or near-100% classification accuracy within fallback or native workflows.
  5. Grok-4 classified almost perfectly but was weaker on data handling autonomy and exploratory depth.
  6. Qwen 3.5, the only open-source model, failed unrecoverably at supervised classification after repeated attempts.

The full story

The study evaluated the models through natural language interaction rather than hand-written code, asking each to carry out the steps of a chemometric workflow: data exploration, unsupervised and supervised classification, model validation and interpretation. The data came from low-field NMR measurements of edible vegetable oils.

Results formed a clear hierarchy. Claude Opus 4.6 ran everything on its own with the highest analytical rigor. Gemini 3.1 Pro and ChatGPT (GPT-5.4 Thinking) were described as effective analytical co-pilots, with accuracy at or near 100% in fallback or native workflows. Grok-4 produced near-perfect classification but needed more handling help and explored the data less deeply.

Qwen 3.5 could not get past supervised classification despite repeated attempts. The authors say the spread in results shows that linguistic fluency is a poor guide to analytical competence and that general benchmarks may not capture domain-specific chemometric performance, so human oversight is still needed.

What this doesn’t fix

The evaluation used a single low-field NMR dataset of edible vegetable oils, and the authors note that general-purpose benchmarks may have limitations for domain-specific chemometric performance.

Sources

1 on file
  1. Analytical Chemistrydoi:10.1021/acs.analchem.6c02701
    Peer-reviewed2026-11-17

Talk it over

Three questions to start on, two more if you have time
Claude Opus 4.6 ran the whole pipeline alone and predicted all nine unlabeled samples, while Qwen 3.5 failed at supervised classification. How would you weigh that spread when picking a tool for lab work?

Start with a question

Only one of five AI chatbots ran a full NMR oil analysis on its own, a study findshttps://truvace.com/item/llms-in-the-laboratory-benchmarking-conversational-ai-for-multivariate-analysis-

  1. Claude Opus 4.6 ran the whole pipeline alone and predicted all nine unlabeled samples, while Qwen 3.5 failed at supervised classification. How would you weigh that spread when picking a tool for lab work?

    Answer in the thread
  2. All five chatbots worked on one low-field NMR dataset of edible vegetable oils. How far could the ranking hold for other oils, other instruments or other kinds of samples?

    Answer in the thread
  3. If you were a chemist using Gemini 3.1 Pro or ChatGPT as an analytical co-pilot, which steps would you check by hand, and why?

    Answer in the thread
If you have more time
  • The authors say fluent language is not a reliable sign of analytical competence. What test could you design to tell a fluent answer from a correct one?

    Answer in the thread
  • Qwen 3.5 is the only open-source model and it failed unrecoverably. What would you want to find out before concluding anything about open-source models in general?

    Answer in the thread

What readers say