Truace
Subscribe

Concept before capability - a validity-led approach to AI in OSCEs

Objective Structured Clinical Examinations (OSCEs) remain central to the assessment of clinical competence, yet the integration of artificial intelligence (AI) into assessment design, delivery, and scoring raises a fundamental question: what evidence of competence are OSCEs intended to generate? This paper argues that AI should be evaluated through the lens of OSCE validity, with particular attention to how it may strengthen or weaken the interpretive chain linking observed performance to assessment decisions. D…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1648· cite & verify
Concept before capability - a validity-led approach to AI in OSCEs
Hospital Universitari Doctor Peset, València 04 by 19Tarrestnom65. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0

In brief

The paper examines how artificial intelligence could be used in designing, delivering and scoring Objective Structured Clinical Examinations, which are central to judging clinical competence. It applies Messick's unitary view of validity and Kane's argument-based framework to ask how AI affects the chain of inferences linking observed performance to assessment decisions.

The significance is that OSCEs inform high-stakes decisions, so tools that look plausible but drift away from the intended skills could undermine trust in results. The paper is a conceptual proposal published in 2026; it does not report measured outcomes, so how the framework performs in real exams remains untested.

Main points

  1. The authors argue AI should be judged by its effect on OSCE validity, not by its technical capability.
  2. AI may introduce threats across the scoring, generalisation, extrapolation and implication inferences.
  3. Generative AI is singled out for the risk of construct drift in assessment materials and scoring.
  4. Design support and post-hoc analytics are treated as lower-risk uses when paired with construct-focused review and human oversight.
  5. AI-mediated candidate encounters and automated summative scoring are treated as higher-risk and need stronger validity evidence.

The problem

Generative AI used in OSCE materials or scoring may gradually reward observable proxies instead of the clinical capabilities the exams are meant to assess, weakening the validity of high-stakes decisions.

The rundown

The authors propose a framework that maps AI applications across the OSCE workflow by how close they sit to candidate consequences and by the evidential burden needed to justify them. This sorts uses into lower-risk and higher-risk categories rather than treating AI as uniformly acceptable or unacceptable.

Their guiding principle, concept before capability, holds that assessment communities must first define what clinical competence means and what evidence should count. Only after that should they decide where AI has a legitimate role in high-stakes assessment.

Sources

  1. Peer-reviewedAdvances in Health Sciences Education2026-10-16

The debate