AI predicted knee arthritis outcomes worse than a doctor in a study of 111 patients

The model also matched the physician's baseline diagnosis in 105 of 111 patients (94.6%), so reading current status from interview records and imaging reports worked far better than forecasting. Its forecasts did not beat a persistence model that simply assumes no change (error 5.43), which limits the case for using it to predict outcomes.

Results were also unstable at patient level. Three independent runs gave highly correlated scores (intraclass correlation 0.982 and 0.965), yet exact agreement between runs was only 31.5% and 6.3% on the two measures. The authors conclude that prognostic uses need evidence of value over a persistence model.

The study used a single-center, prospective observational cohort with a blinded, retrospective evaluation of the language model. Both DeepSeek-V4-Flash and a rheumatologist worked from the same standardized interview records and imaging reports, while a separate senior physician determined the actual outcomes used as the reference.

The comparison had three arms: the model, the physician and a persistence model that assumes no change. The physician had the lowest WOMAC error (3.86) and highest Kellgren-Lawrence exact agreement (86.5%), while the model's errors were higher than the persistence model's.

Reproducibility was mixed. Intraclass correlation coefficients across three runs were high (0.982 and 0.965), but patient-level exact agreement between runs was 31.5% and 6.3%. The authors say the configuration may support structured baseline assessment, while its predictive accuracy and stability remain limited.