Emerging evidence
More
Residents felt more confident judging AI output after a 30-minute workshop, a survey of 97 responses finds
Study title: Beyond the Black Box: A 30-Minute Artificial Intelligence Critical Appraisal Workshop for Internal Medicine Residents
- Evidence type
- Survey
- Sample
- n = 97
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-3012cite or check it
Study title: Beyond the Black Box: A 30-Minute Artificial Intelligence Critical Appraisal Workshop for Internal Medicine Residents
A 30-minute workshop on critically appraising AI output replaced one evidence-based medicine journal club session for postgraduate year 1 to 3 internal medicine residents. Six sessions ran from July to December 2025. Residents prompted a large language model with a fictional patient scenario, compared outputs in small groups and learned a structured appraisal framework.
Read the study abstract
Introduction: Artificial intelligence (AI) and large language models (LLMs) are rapidly entering resident workflows. However, formal training on appropriate LLM use is limited, and AI outputs may be inaccurate, outdated, or biased. This creates a curricular gap: residents may use AI before receiving guidance on evaluating AI-generated recommendations. Methods: We developed a 30-minute AI critical appraisal workshop for postgraduate year 1 to 3 (PGY-1-PGY-3) internal medicine residents. The workshop replaced 1 ev…

The gain
In an anonymous before-and-after survey at one internal medicine program, residents' confidence in critically appraising AI output rose from 60.4% before a 30-minute workshop to 79.5% after it.
In brief
Of 72 eligible residents, 53 completed a pre-workshop survey and 44 a post-workshop survey. Confidence in appraising AI output rose from 60.4% to 79.5%, and 90.5% rated the session 4 or 5 out of 5.
Concern about inaccurate AI predictions rose from 69.8% to 88.6%, and concern about bias from 77.4% to 88.6%. Belief that AI improves health outcomes stayed high at 94.3% before and 97.7% after. Surveys were anonymous and unpaired, so the results were analyzed descriptively only.
Main points
- The workshop lasted 30 minutes and replaced one evidence-based medicine journal club session during ambulatory continuity clinic blocks.
- Six workshops were delivered between July and December 2025 to PGY-1 to PGY-3 internal medicine residents.
- Residents prompted an LLM with a fictional patient scenario, compared outputs in small groups and learned a structured appraisal framework.
- Of 72 eligible residents, 53 completed the pre-workshop survey and 44 the post-workshop survey.
- Self-reported confidence in appraising AI output rose from 60.4% to 79.5%.
- Concern about inaccurate AI predictions rose from 69.8% to 88.6%, and concern about AI bias from 77.4% to 88.6%.
- Belief that AI improves health outcomes stayed high, at 94.3% before and 97.7% after, and 90.5% rated the session 4 or 5 out of 5.
The full story
The authors describe a curricular gap: residents may use AI tools before getting guidance on evaluating AI-generated recommendations, even though outputs can be inaccurate, outdated or biased. The workshop was designed to fill that gap within existing teaching time.
Surveys were collected anonymously through QR codes immediately before and after each session. Because responses were unpaired, the changes compare two groups of respondents rather than the same residents over time.
Residents became more worried about inaccurate predictions and bias after the session while their belief that AI improves health outcomes remained high. The authors conclude the required workshop was feasible and well received, and that it is adaptable, needs minimal technology and fits into existing evidence-based medicine teaching.
Sources
1 on file- MedEdPORTALdoi:10.15766/mep_2374-8265.11654
Talk it over
Residents' confidence in appraising AI output rose from 60.4% to 79.5%, while concern about inaccurate AI predictions rose from 69.8% to 88.6%. How would you weigh the higher confidence against the higher concern?
Residents felt more confident judging AI output after a 30-minute workshop, a survey of 97 responses findshttps://truvace.com/item/beyond-the-black-box-a-30-minute-artificial-intelligence-critical-appraisal-work
Residents' confidence in appraising AI output rose from 60.4% to 79.5%, while concern about inaccurate AI predictions rose from 69.8% to 88.6%. How would you weigh the higher confidence against the higher concern?
Answer in the threadThe surveys were anonymous and unpaired, with 53 residents answering before and 44 after. How does comparing two groups of respondents, rather than the same residents, change what the confidence figures can tell us?
Answer in the threadOf 72 eligible residents, some did not answer either survey. Who might those residents be, and how could their answers change the 60.4% and 79.5% figures?
Answer in the thread
If you have more time
What would you want to ask residents several months after the 30-minute workshop to learn whether the structured appraisal framework changed how they judge AI output with real patients?
Answer in the threadIf a hospital you know replaced one journal club session with a workshop like this, what would you track to decide whether it was worth the swap?
Answer in the thread
ace

What readers say