Truace
Subscribe

Evaluation and Comparison of Large Language Models in Answering Frequently Asked Patient Questions on Glaucoma

Prcis: ChatGPT-o1 and DeepSeek-R1 demonstrated similar performance in answering frequently asked patient questions. However, limitations in reliability and high reading levels underscore the need for greater oversight and refinement to improve their role in patient education. Purpose: This study evaluated and compared the reliability and readability of responses generated by ChatGPT-o1 and DeepSeek-R1 to frequently asked patient questions (FAQs) on glaucoma. The findings aim to elucidate the utility of each mode…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1380· cite & verify
Evaluation and Comparison of Large Language Models in Answering Frequently Asked Patient Questions on Glaucoma
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

In brief

Researchers took the first 10 glaucoma questions from Google's People Also Ask section and put them to two chatbots, ChatGPT-o1 and DeepSeek-R1. Two glaucoma specialists graded each answer on a 4-point reliability scale, and the top two Google webpages for the same queries were graded in the same way. Readability was measured with validated metrics.

Patients increasingly look for health information online, so errors or hard-to-read answers from chatbots could affect how well they understand a chronic eye disease. The authors call for oversight and refinement of these models. The study covers only 10 questions and two models, so how far the findings extend is unclear.

Main points

  1. Median reliability scores were 2.3 for ChatGPT-o1 and 2.8 for DeepSeek-R1, with no significant difference (P=0.08).
  2. Top Google webpages scored 2.3, similar to the chatbots (P=0.18).
  3. Both chatbots struggled with foundational knowledge and lifestyle questions.
  4. Chatbot responses were written at the 8th-12th grade reading level, above average U.S. adult literacy.

The problem

ChatGPT-o1 and DeepSeek-R1 gave glaucoma patient answers with notable accuracy errors, including false claims about tunnel vision and unsubstantiated lifestyle advice, written at 8th-12th grade reading levels.

The rundown

The questions covered foundational knowledge, treatment description, lifestyle, and prognosis. Each was entered into both models, and two glaucoma specialists independently graded the responses, where 1 meant excellent with no clarification needed and 4 meant unsatisfactory.

The chatbots made specific errors: they falsely suggested glaucoma may cause tunnel vision, and they offered unsubstantiated advice on chugging water, diet, and screen time.

The authors conclude that the two models performed comparably and that their reading levels exceed average U.S. adult literacy, which poses challenges for patient comprehension. They say rigorous oversight and targeted refinement are needed before these tools are relied on for patient education.

Sources

  1. Peer-reviewedJournal of Glaucoma2026-10-12

The debate