Truace
Subscribe

Reliability, quality, and readability of large language models in carpal tunnel syndrome: A comparative cross-linguistic, cross-sectional observational study

BackgroundAI-based chatbots are increasingly used for health information, while cross-linguistic studies suggest that response performance may vary by language. Given the high prevalence of carpal tunnel syndrome (CTS), this study evaluated the reliability, quality, patient education suitability, and readability of AI-generated responses to CTS questions in English and Turkish.Materials and methodsSeventeen patient-centered CTS questions were directed to ChatGPT ® (GPT-5 Mini), Gemini ® 3 Flash, and Claude ® 4.5…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1730· cite & verify
Reliability, quality, and readability of large language models in carpal tunnel syndrome: A comparative cross-linguistic, cross-sectional observational study
Carpal Tunnel by Maen K. Househ. CC BY-SA 3.0 · https://creativecommons.org/licenses/by-sa/3.0

In brief

Researchers put seventeen patient-style questions about carpal tunnel syndrome to ChatGPT (GPT-5 Mini), Gemini 3 Flash and Claude 4.5 Sonnet. Each question was asked in English and Turkish across 102 independent sessions, and the answers were scored for reliability, quality, patient-education suitability and readability.

The findings matter because patients increasingly turn to chatbots for health information, and answers that are hard to read or uneven across languages may limit their usefulness. The authors recommend treating chatbots as supplementary sources only. The abstract does not directly measure factual clinical accuracy, so how correct the advice was remains unresolved.

Main points

  1. The study compared three chatbots: ChatGPT (GPT-5 Mini), Gemini 3 Flash and Claude 4.5 Sonnet.
  2. Seventeen patient-centered questions were asked in English and Turkish using zero-shot prompting across 102 independent sessions.
  3. Responses were scored with mDISCERN, Global Quality Score and the Patient Education Materials Assessment Tool.
  4. Readability was measured with the Flesch Reading Ease for English and the Ateşman Index for Turkish.
  5. Significant differences between models were found in English across all metrics.

The problem

None of the three tested chatbots produced carpal tunnel syndrome answers at recommended readability levels in English or Turkish, and quality varied by model and language.

The rundown

The design was a cross-sectional observational comparison. The statistical analysis used Friedman and Wilcoxon tests with Bonferroni correction to compare models and languages.

Reliability was assessed with mDISCERN, quality with the Global Quality Score, and patient-education suitability with PEMAT. In English, significant differences between models appeared on every one of these metrics.

The authors caution that mDISCERN partly reflects whether references are provided, so score differences should be read carefully. They conclude that chatbots should supplement, not replace, other sources of carpal tunnel information for patients.

Sources

  1. Peer-reviewedJournal of Back and Musculoskeletal Rehabilitation2026-10-08

The debate