Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2290
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-27T18:35:10.152049Z
status: published
lens: p_space
sector: health
headline: Development and Pilot Evaluation of a Standardized and Accessible Rubric (OMNI) for Assessing Generative AI in Health Care Contexts: Four-Stage Mixed Methods Study
dek: Background: The rapid adoption of generative AI in the community and health care necessitates reliable, human-led evaluation.
Objective: This study aimed to develop a standardized rubric to support health care professionals, administrators, and researchers in evaluating generative AI outputs in health contexts.
Methods: A rubric was developed through a 4-stage mixed methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 69 publications and r…
gain_title: (none)
problem_title: When 15 clinicians rated generative AI outputs across four health scenarios, recurring weaknesses appeared in referencing, recognizing uncertainty, completeness, empathy and accuracy, and image and video outputs fared worst.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: When 15 clinicians rated generative AI outputs across four health scenarios, recurring weaknesses appeared in referencing, recognizing uncertainty, completeness, empathy and accuracy, and image and video outputs fared worst.
problem_evidence: recurrent limitations in Referencing, Recognizes Uncertainty, Completeness, Empathetic, and Accuracy were identified across scenarios | Multimodal image and video outputs had the highest occurrence of negative rating domains
quick_read: Researchers built a ten-domain rubric called OMNI for health professionals, administrators and researchers to judge generative AI outputs. They drew on 69 publications and a 23-member expert panel, then tested it in four health care scenarios.

The work matters because health care is adopting generative AI quickly and needs consistent, human-led ways to check it. The pilot showed the tool can separate model strengths and weaknesses, but rater agreement was modest on the full scale, and its fit for purpose-built tools and agentic systems is untested.
limitation: Further work is needed to assess whether OMNI applies to purpose-built health AI tools, agentic systems and organizational evaluation settings, and rater agreement on the 4-point scales was modest.
tag: Evidence-backed problem
key_points: OMNI has 10 domains, including Accuracy, Safety, Recognizes Uncertainty and Avoids Coerciveness, scored on 4-point Likert scales. | 15 clinicians evaluated 250 input-output pairs, producing 9600 domain-level ratings and 4010 qualitative comments. | Mean pairwise agreement was 30.8% to 53.4% on the 4-point scales and 59.3% to 79.6% when collapsed into broad categories. | Assessors said OMNI was usable and that they would reuse it.
rundown: The rubric was developed in four stages: literature synthesis, expert refinement, pilot application and post-pilot feedback. The expert panel had 23 members, and the evidence base covered 69 publications.

The pilot covered responses to health-related Reddit questions, generated health images and videos, answers to continuing education questions, and automated physician-to-physician clinical documentation. The 250 pairs included 200 AI responses and 50 physician-tagged Reddit replies.

Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses between models within scenarios while generating structured feedback.
sources:
- peer_reviewed | Journal of Medical Internet Research | https://doi.org/10.2196/100531 | 2026-10-27
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

93facf8a57d794b75b71d60ac29d5e98c4e2f384c606a8a5cbba6855851a76a0

previous

0000000000000000000000000000000000000000000000000000000000000000