Truace
Subscribe

Beyond Autonomy: Ethical Considerations of AI in Substitute Decision-Making for Incapacitated Patients

The integration of artificial intelligence (AI) into healthcare offers significant potential for enhancing patient care, particularly for incapacitated patients unable to communicate their preferences. Advanced AI systems, such as Personalized Patient Preference Predictors (P4), are increasingly proposed as tools to assist in substitute decision-making, yet their ethical implications remain underexamined. This paper critically examines the ethical implications of employing AI-driven preference prediction systems…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1403· cite & verify
Beyond Autonomy: Ethical Considerations of AI in Substitute Decision-Making for Incapacitated Patients
Eva Weber-Guskgar - How to feel about emotionalized artificial intelligence by Eva Weber‑Guskar. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0

In brief

A paper in the Journal of Evaluation in Clinical Practice examines AI systems called Personalized Patient Preference Predictors, which are proposed to help decide for patients who cannot communicate their wishes. It tests them against four ethical principles: autonomy, beneficence, non-maleficence and justice.

The analysis concludes that these tools may augment preference prediction but should not replace the judgment of human surrogates. The significance is that clinical teams may be tempted to treat a prediction as the patient's voice. The source presents this as an ethical analysis, so how the tools perform in real clinical settings remains untested here.

Main points

  1. The paper evaluates P4 systems against autonomy, beneficence, non-maleficence and justice.
  2. It distinguishes probabilistic preference inference from genuine autonomy.
  3. Bias can persist even in highly individualized datasets.
  4. AI is framed as a decision-support tool that augments human surrogates rather than replacing them.
  5. Recommendations include meaningful human oversight, transparent and interpretable AI, and a robust regulatory framework.

The problem

Personalized Patient Preference Predictors may infer probabilities but cannot capture the relational, contextual and value-laden nature of real patient preferences, leaving risks of algorithmic paternalism and latent bias.

The rundown

The paper starts from the premise that AI could improve care for incapacitated patients who cannot state their preferences. Systems such as P4 are increasingly proposed for substitute decision-making, but their ethical implications are described as underexamined.

Using principled ethical reasoning, the authors identify three main challenges: the gap between predicted preferences and genuine autonomy, algorithmic paternalism even in personalized models, and subtle bias within highly individualized datasets.

The authors propose a framework that places AI predictions within a model of relational autonomy, with human surrogates contextualizing the AI-generated data. Their recommendations are meaningful human oversight, transparency and interpretability, and a regulatory framework.

Sources

  1. Peer-reviewedJournal of Evaluation in Clinical Practice2026-10-01

The debate