Truace
Subscribe

ChatGPT scored occupational therapy students' patient interviews fairly consistently

Study title: Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs

Researchers at a university in northern Taiwan built a ChatGPT-based scoring tool called G-ACE for chief-complaint interview exams in occupational therapy. They tested it on 58 de-identified transcripts of 8-minute interviews between undergraduate students and a simulated stroke patient.

Read the study abstract

ImportanceConversation-related clinical skills are central to Objective Structured Clinical Examinations (OSCEs) in occupational therapy. However, real-time examiner scoring is resource-intensive and may overlook students' specific skills. Transcript-based scoring with Chat Generative Pre-Trained Transformer (ChatGPT) may address these limitations.ObjectiveTo develop a ChatGPT-based assessment for chief-complaint interview OSCEs (G-ACE) and examine criterion validity against expert-consensus ratings, test-retest…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 64/100 — Controlled or validated comparison · n = 58 · 1 independent sourcePeer-reviewedTRV-2026-2353· cite & verify
Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs
PHOTOS INSIDE THE CLASSROOM UPDATED005 by Monirb. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0

In brief

Automated transcript scoring could ease the burden of real-time examiner scoring and catch skills examiners might miss. Agreement with experts was uneven across items, so the tool is not yet a full replacement, and its feedback value needs careful interpretation.

Main points

  1. Fifty-eight transcripts from undergraduate occupational therapy students were analysed in a secondary analysis.
  2. Agreement with expert consensus was acceptable for primary symptoms and impact on daily life items (weighted kappa 0.60 or higher).
  3. Rehabilitation expectations and communication skills items agreed less well with experts (0.18-0.59).
  4. Domain and total score test-retest reliability reached ICC 0.65-0.85.

The gain

A ChatGPT-based tool scored student interview transcripts with acceptable repeat-scoring reliability and small absolute differences from expert ratings on domain and total scores, offering an alternative to resource-intensive real-time examiner scoring.

The rundown

The study used a simulated patient with stroke in an OSCE setting, and ChatGPT scored the interview transcripts rather than examiners watching in real time. Criterion validity was judged against expert-consensus ratings.

Item-level test-retest reliability was weighted kappa 0.68-1.00, and domain and total scores had mean absolute errors of 2.3%-9.1% on repeat scoring. Compared with experts, domain and total score errors ranged from 4.6% to 20.7%.

Inter-rater reliability was acceptable for most domain and total scores (ICCs 0.52-0.78), but the rehabilitation expectations domain was weak (ICCs 0.17-0.30).

Sources

  1. Peer-reviewedAmerican Journal of Occupational Therapy2026-11-03

The debate