Truace
Subscribe

Chatbot answers to patient questions about glenohumeral osteoarthritis

Source article: Evaluating the Reliability, Accuracy, and Readability of ChatGPT-3.5 and GPT-4 in Providing Patient Education Related to Glenohumeral Joint Osteoarthritis

Objectives Chatbots have been increasingly recognized as modern tools that provide patients with reliable health-related information. This study aimed to evaluate and compare ChatGPT-3.5 and GPT-4's ability to answer glenohumeral osteoarthritis-related questions. Methods Fifteen questions were derived from the 2020 AAOS Clinical Practice Guidelines for the Surgical Management of Glenohumeral Joint Osteoarthritis. Questions were categorized into three groups: risk factors, implant/intraoperative considerations, a…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1515· cite & verify
Evaluating the Reliability, Accuracy, and Readability of ChatGPT-3.5 and GPT-4 in Providing Patient Education Related to Glenohumeral Joint Osteoarthritis
Flickr - Official U.S. Navy Imagery - Sgt. Joshua Elliott uses a Primus RS machine during physical therapy. by Official Navy Page from United States of America MCSN Joseph A. Boomhower/U.S. Navy. Public domain
Trace impact readingContested
P 68The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers wrote fifteen questions based on the 2020 AAOS guidelines for surgical management of glenohumeral osteoarthritis and put them to ChatGPT-3.5 and GPT-4. Four fellowship-trained shoulder and elbow surgeons then rated every response from 1 to 5 on five criteria, and readability was measured on the Flesch-Kincaid scale.

The findings matter because patients increasingly consult chatbots for health information, and this study suggests the answers are generally adequate but not strongly grounded in evidence. The abstract points to better literature-based content and more accessible wording as needs for future versions; it does not report how patients themselves would use or understand the answers.

Main points

  1. Fifteen questions were drawn from the 2020 AAOS Clinical Practice Guidelines and grouped into risk factors, implant/intraoperative considerations, and pain/functional outcomes.
  2. ChatGPT-3.5 averaged 19.7/25, with risk-factor prompts scoring highest.
  3. GPT-4 averaged 18.7/25, with functional-outcome prompts scoring highest.
  4. Neither model showed statistically significant differences between prompt themes.
  5. Flesch-Kincaid scores were 18.3 for GPT-3.5 and 15.4 for GPT-4, so GPT-3.5 was harder to read.

The gain

Both chatbots gave adequately well-informed answers to patient-style questions about glenohumeral osteoarthritis, as judged by four fellowship-trained shoulder surgeons.

The problem

Both models scored lowest on evidence-based content, and GPT-3.5 answers were written at a harder reading level than GPT-4's.

The rundown

Each response was rated by four surgeons for relevance, accuracy, clarity, completeness, and evidence-based support, giving a maximum of 25 points per question. Data were analyzed descriptively and statistically to compare the two models.

Strengths differed by model: clarity was the highest-scoring category for GPT-3.5, while relevance was highest for GPT-4. Evidence-based support was the lowest-scoring category for both.

The authors conclude that future chatbot versions should draw on systematic and reliable literature reviews and present content in an accessible, readable way.

What this doesn’t fix

The study used only fifteen guideline-derived questions on one condition, and the abstract reports no statistically significant differences between prompt themes, which limits how far the results generalize.

Sources

  1. Peer-reviewedRhode Island medical journal (2013)2026-10-01

The debate