30/100

Emerging evidence

Cite
More
EmailLinkedInX

A new metric would score medical AI by how risky its errors are, researchers propose

Study title: Normalized risk-based evaluation of machine learning-based classification models: A multiclass approach with applications in medical devices

Evidence type
Commentary, opinion, or narrative review
Sources
1 sourcePeer-reviewed
Published
1 min read

Study title: Normalized risk-based evaluation of machine learning-based classification models: A multiclass approach with applications in medical devices

Researchers propose weighted balanced accuracy (WBAn) to evaluate multiclass machine learning models used in medical devices, aiming to align model assessment with regulatory risk management. They demonstrate the metric using X-ray based detection of lung diseases as a reference scenario.

Read the study abstract

ObjectiveMachine learning is increasingly integrated into safety-critical domains such as medical applications. In this context, regulatory frameworks require the assessment and minimization of risks associated with incorrect model predictions. However, classical evaluation methods often focus on quantifying error frequencies, which do not reflect the heterogeneous impact of different types of errors. To address this deficiency, we elaborate an approach for assessing machine learning-based multiclass classificat…

Normalized risk-based evaluation of machine learning-based classification models: A multiclass approach with applications in medical devices
Hospital Universitari Doctor Peset, València 10 by 19Tarrestnom65. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0

The gain

Weighted balanced accuracy (WBAn) provides a regulatory-aligned, risk-based evaluation for multiclass medical AI that weights errors by severity and links development to real-world performance, demonstrated in X-ray lung disease detection.

The problem

Standard evaluation that counts error frequencies fails to reflect differing clinical severity of error types, leaving regulatory risk requirements incompletely implemented and real-world clinical impact inadequately assessed.

In brief

The work matters because current accuracy measures treat all errors equally, while medical errors carry different clinical severity. By weighting errors by risk, WBAn could improve regulatory compliance and real-world safety assessment, though evidence is limited to one illustrative use case as of the September 2026 publication.

Main points

  1. Authors propose weighted balanced accuracy WBAn as a risk-based metric for multiclass classification in medical devices.
  2. WBAn operationalizes risk as a multiplicative combination of likelihood and severity per regulatory requirements.
  3. Demonstration use case is X-ray based detection of lung diseases as a reference scenario.
  4. Paper argues classical error-frequency metrics do not capture heterogeneous clinical impact of different error types.

The full story

The methods section describes WBAn as systematically based on regulatory requirements for medical devices, incorporating risk as a multiplicative combination of likelihood and severity and the relationship between development and real-world scenarios.

Results and conclusion report that WBAn achieves risk-based assessment in the lung disease reference and that without such an approach the clinical impact cannot be adequately assessed when applying the model in real-world scenarios.

Sources

1 on file
  1. Peer-reviewed2026-09-28

Talk it over

What readers make of this record

What readers say