Truace
Subscribe

Clinicians test new rubric that finds repeated weak spots in AI health answers

Study title: Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study

Researchers built a ten-domain rubric called OMNI for health professionals, administrators and researchers to judge generative AI outputs. They drew on 69 publications and a 23-member expert panel, then tested it in four health care scenarios.

Read the study abstract

Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation. Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts. Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and r…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 68/100 — Controlled or validated comparison · n = 250 · 1 independent sourcePeer-reviewedTRV-2026-2290· cite & verify
Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study
Emoji Use in the Electronic Health Record by David A. Hanauer, Gavin C. Raab, Shira N. Hanauer, Lisa Ferguson, Kellen McClain, Guan Wang, Michelle Rozwadowski, Sung W. Choi. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0

In brief

The work matters because health care is adopting generative AI quickly and needs consistent, human-led ways to check it. The pilot showed the tool can separate model strengths and weaknesses, but rater agreement was modest on the full scale, and its fit for purpose-built tools and agentic systems is untested.

Main points

  1. OMNI has 10 domains, including Accuracy, Safety, Recognizes Uncertainty and Avoids Coerciveness, scored on 4-point Likert scales.
  2. 15 clinicians evaluated 250 input-output pairs, producing 9600 domain-level ratings and 4010 qualitative comments.
  3. Mean pairwise agreement was 30.8% to 53.4% on the 4-point scales and 59.3% to 79.6% when collapsed into broad categories.
  4. Assessors said OMNI was usable and that they would reuse it.

The problem

When 15 clinicians rated generative AI outputs across four health scenarios, recurring weaknesses appeared in referencing, recognizing uncertainty, completeness, empathy and accuracy, and image and video outputs fared worst.

The rundown

The rubric was developed in four stages: literature synthesis, expert refinement, pilot application and post-pilot feedback. The expert panel had 23 members, and the evidence base covered 69 publications.

The pilot covered responses to health-related Reddit questions, generated health images and videos, answers to continuing education questions, and automated physician-to-physician clinical documentation. The 250 pairs included 200 AI responses and 50 physician-tagged Reddit replies.

Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios while generating structured feedback.

Sources

  1. Peer-reviewedJournal of Medical Internet Research2026-10-27

The debate