42/100

Emerging evidence

Cite
More
EmailLinkedInX

Doctors in a 20-physician survey would act on false AI-written abstracts

Study title: Hallucinated Evidence and Real-World Risk-Physicians' Intent to Act on AI-Generated Abstracts in Emergency Medicine and Trauma Surgery: Cross-Sectional, Vignette-Based Diagnostic Accuracy Study

Evidence type
Survey
Sample
n = 20
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Hallucinated Evidence and Real-World Risk-Physicians' Intent to Act on AI-Generated Abstracts in Emergency Medicine and Trauma Surgery: Cross-Sectional, Vignette-Based Diagnostic Accuracy Study

Twenty emergency medicine and trauma surgery physicians at 4 hospitals in Germany each rated 4 abstracts written with GPT-5 Thinking mode. Two abstracts held deliberately false claims backed by plausible explanations; two held accurate claims. Across 80 ratings, physicians said they would act on 68% (54/80) of the abstracts.

Read the study abstract

Background: Large language models (LLMs) are increasingly used in academic writing and can generate linguistically polished but factually incorrect scientific content. Such misinformation may distort evidence appraisal and, if translated into clinical practice, compromise patient care. Objective: This study aimed to estimate physicians' likelihood of misperceiving false information embedded in LLM-generated abstracts as true and reporting an intent to use that information in future practice. We also explored whi…

The problem

In a vignette survey of 20 emergency and trauma physicians at 4 German hospitals, doctors said they would act on 57% (31/54) of the abstracts they chose to act on that were GPT-5-generated fakes, and rated true and false abstracts similarly for accuracy.

In brief

Intent to act barely separated truth from falsehood. Sensitivity was 78% and specificity 43%, so more than half of the false abstracts (23 of 40) drew a stated intent to act. The positive likelihood ratio of 1.35 had a confidence interval (0.98-1.85) reaching down to 1.

The result matters because LLMs are increasingly used in academic writing and can produce polished but wrong content. The sample was small and the task was hypothetical, so the study measures stated intent rather than actual clinical decisions.

Main points

  1. Design: cross-sectional vignette survey of physicians in emergency medicine and trauma surgery at 4 hospitals in Germany.
  2. Twenty physicians gave 80 ratings; each assessed 4 GPT-5 Thinking abstracts, 2 false and 2 accurate.
  3. Physicians said they would act on 68% (54/80) of abstracts.
  4. Using truth status as the reference, acting had 78% sensitivity (31/40), 43% specificity (17/40), positive predictive value 57% and negative predictive value 65%.
  5. Perceived accuracy was similar for true and false abstracts (mean 3.15 vs 2.88; P=.21).
  6. Abstracts physicians would act on scored higher for accuracy, precision and structure (all P<.001); grammar did not differ (P=.80).

The full story

The study tested whether LLM-generated misinformation could pass as credible to practising clinicians. Each participant read four GPT-5 Thinking abstracts, half with intentionally false claims and half accurate. The main outcome was the proportion of doctors reporting an intent to use or act on recommendations in the false abstracts.

Diagnostic-accuracy measures showed weak discrimination: positive predictive value was 57% (31/54) and negative predictive value 65% (17/26). The diagnostic odds ratio was 2.55 (95% CI 0.96-6.73), with intervals that include no discrimination.

Ratings of precision (3.15 vs 2.93; P=.29), structure (3.23 vs 3.20; P=.89) and grammar (3.40 vs 3.43; P=.86) were also similar for true and false abstracts. Yet abstracts that doctors would act on scored substantially higher on perceived accuracy, precision and structure, suggesting that polished presentation drives credibility.

Sources

1 on file
  1. Peer-reviewed2026-11-13

Talk it over

Three questions to start on, two more if you have time
Physicians said they would act on 23 of 40 false abstracts and rated true and false ones similarly for accuracy. Why might polished presentation have shaped their judgment more than the claims did?

Start with a question

Doctors in a 20-physician survey would act on false AI-written abstractshttps://truvace.com/item/hallucinated-evidence-and-real-world-risk-physicians-intent-to-act-on-ai-generat

  1. Physicians said they would act on 23 of 40 false abstracts and rated true and false ones similarly for accuracy. Why might polished presentation have shaped their judgment more than the claims did?

    Answer in the thread
  2. The study involved 20 emergency and trauma physicians at 4 hospitals in Germany and measured stated intent, not real decisions. How might actual clinical practice change what these doctors did with the abstracts?

    Answer in the thread
  3. If you were one of these physicians handed four abstracts, what would you check before saying you would act on any of them?

    Answer in the thread
If you have more time
  • Which other groups, such as doctors in other specialties or countries, would you want tested with the same 2 false and 2 accurate abstracts?

    Answer in the thread
  • What result would show that doctors can tell the GPT-5 Thinking fakes from accurate abstracts, given the likelihood ratio of 1.35?

    Answer in the thread

What readers say