Truace
Subscribe

Performance of Large Language Models in Answering Juvenile Systemic Lupus Erythematosus Questions: A Blinded Expert-based Comparative Study

Background Large language models are increasingly used to obtain medical information, but their performance in juvenile systemic lupus erythematosus remains insufficiently characterized. Objective To compare expert-rated response quality and clinical relevance across 4 artificial intelligence (AI) models using standardized juvenile systemic lupus erythematosus-related questions. Design Cross-sectional expert-based comparative study. Setting Freely accessible consumer-facing web interfaces evaluated on December 3…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1701· cite & verify
Performance of Large Language Models in Answering Juvenile Systemic Lupus Erythematosus Questions: A Blinded Expert-based Comparative Study
"Senator Stabenow meets with members of the American College of Rheumatology." by Senator Stabenow is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.

In brief

Ten pediatric rheumatology experts rated responses from four consumer-facing AI models to twenty standardized questions about juvenile systemic lupus erythematosus, a total of 80 answers, on a 5-point scale. The models were tested through freely accessible web interfaces on December 30, 2025.

The results matter because people increasingly turn to chatbots for medical information, and the models did not perform equally on a complex pediatric autoimmune condition. How reliable the answers are over repeated queries is still unknown, so the findings support cautious use with specialist oversight rather than clinical reliance.

Main points

  1. DeepSeek had the highest median expert rating, 5 on a scale with a range of 1 to 5.
  2. Gemini, ChatGPT and Copilot had median ratings of 4, with differing score ranges.
  3. Overall performance differed significantly among the four models in a Friedman test.
  4. Interrater agreement was assessed with intraclass correlation coefficients, and the authors report it differed across models.

The problem

Expert-rated quality of chatbot answers on juvenile lupus differed significantly across models, so outputs need caution and specialist oversight.

The rundown

The study was a cross-sectional, expert-based comparison. Twenty standardized juvenile lupus questions were put to four AI models, and ten pediatric rheumatology experts blindly rated the 80 responses for quality and clinical relevance.

Statistical analysis used the Friedman test with Kendall W and Bonferroni-corrected post hoc comparisons. Median ratings were DeepSeek 5 (range 1 to 5), Gemini 4 (2 to 5), ChatGPT 4 (3 to 5) and Copilot 4 (3 to 5).

The authors conclude that the models differed in expert-rated performance and in agreement among raters, and that outputs should be read cautiously and with specialist oversight.

Sources

  1. Peer-reviewedJCR: Journal of Clinical Rheumatology2026-10-08

The debate