Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2353
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-03T18:02:05.792950Z
status: published
lens: g_space
sector: health
headline: Development and Validation of a ChatGPT-Based Assessment for Conversation-Related Clinical Skills in Occupational Therapy OSCEs
dek: ImportanceConversation-related clinical skills are central to Objective Structured Clinical Examinations (OSCEs) in occupational therapy. However, real-time examiner scoring is resource-intensive and may overlook students' specific skills. Transcript-based scoring with Chat Generative Pre-Trained Transformer (ChatGPT) may address these limitations.ObjectiveTo develop a ChatGPT-based assessment for chief-complaint interview OSCEs (G-ACE) and examine criterion validity against expert-consensus ratings, test-retest…
gain_title: A ChatGPT-based tool scored student interview transcripts with acceptable repeat-scoring reliability and small absolute differences from expert ratings on domain and total scores, offering an alternative to resource-intensive real-time examiner scoring.
problem_title: (none)
trace_subject: (none)
gain_reading: A ChatGPT-based tool scored student interview transcripts with acceptable repeat-scoring reliability and small absolute differences from expert ratings on domain and total scores, offering an alternative to resource-intensive real-time examiner scoring.
gain_evidence: small absolute differences from expert ratings for domain and total scores
problem_reading: (none)
problem_evidence: (none)
quick_read: Researchers at a university in northern Taiwan built a ChatGPT-based scoring tool called G-ACE for chief-complaint interview exams in occupational therapy. They tested it on 58 de-identified transcripts of 8-minute interviews between undergraduate students and a simulated stroke patient.

Automated transcript scoring could ease the burden of real-time examiner scoring and catch skills examiners might miss. Agreement with experts was uneven across items, so the tool is not yet a full replacement, and its feedback value needs careful interpretation.
limitation: Some items showed lower consistency with expert ratings, and the rehabilitation expectations domain had weak inter-rater reliability, so formative feedback should be interpreted cautiously.
tag: Evidence-backed gain
key_points: Fifty-eight transcripts from undergraduate occupational therapy students were analysed in a secondary analysis. | Agreement with expert consensus was acceptable for primary symptoms and impact on daily life items (weighted kappa 0.60 or higher). | Rehabilitation expectations and communication skills items agreed less well with experts (0.18-0.59). | Domain and total score test-retest reliability reached ICC 0.65-0.85.
rundown: The study used a simulated patient with stroke in an OSCE setting, and ChatGPT scored the interview transcripts rather than examiners watching in real time. Criterion validity was judged against expert-consensus ratings.

Item-level test-retest reliability was weighted kappa 0.68-1.00, and domain and total scores had mean absolute errors of 2.3%-9.1% on repeat scoring. Compared with experts, domain and total score errors ranged from 4.6% to 20.7%.

Inter-rater reliability was acceptable for most domain and total scores (ICCs 0.52-0.78), but the rehabilitation expectations domain was weak (ICCs 0.17-0.30).
sources:
- peer_reviewed | American Journal of Occupational Therapy | https://doi.org/10.1177/19437676261500028 | 2026-11-03
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

85c88c962fde803af276eac4174c81a738ae63e495bd4087a86189e2bd020094

previous

0000000000000000000000000000000000000000000000000000000000000000