Truace
Subscribe

Responses of large language models to common questions on premature ovarian insufficiency: A comparative analysis

Objective: Large language models are widely used in clinical care. Their reliability in specialized gynecological diseases such as premature ovarian insufficiency (POI) has not been fully verified. This study aimed to systematically evaluate and compare the accuracy and completeness of three mainstream large language models when answering standardized clinical questions about POI. Methods: We developed 34 standardized questions covering five core domains of POI according to international clinical guidelines. Two…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1783· cite & verify
Responses of large language models to common questions on premature ovarian insufficiency: A comparative analysis
"McArdle Laboratory for Cancer Research" by wackybadger is licensed under CC BY-SA 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/2.0/.

In brief

Researchers built 34 standardized questions across five core domains of premature ovarian insufficiency, drawn from international clinical guidelines. They put the questions to three large language models and had two senior obstetricians and gynecologists independently score the answers on a 5-point scale for accuracy and completeness.

The results suggest these chatbots can handle a specialized gynecological topic reasonably well, which matters as such tools are increasingly used in clinical care. The authors still require qualified clinicians to check all AI-generated content before clinical use, and the test covers only a fixed question set rather than real patient interactions.

Main points

  1. Accuracy scores across the three models ranged from 4.59 to 4.84, and completeness scores from 4.43 to 4.65.
  2. Exact agreement between the two raters was 91.18% to 97.06%, with Cohen's kappa between 0.675 and 0.915.
  3. GPT-5.5 did better on clinical manifestations and treatment questions, while DeepSeek-R1 was more stable on etiology and counseling questions.
  4. Most item-level differences between models fell within ±0.5 points, with no obvious systematic bias.

The gain

Two gynecologists rated GPT-5.5, Claude Opus-4.7 and DeepSeek-R1 as accurate and complete on 34 guideline-based POI questions, supporting supervised use for information retrieval and patient education.

The rundown

The study tested GPT-5.5, Claude Opus-4.7 and DeepSeek-R1 on 34 standardized questions covering five core POI domains. Inter-rater reliability was assessed with Cohen's kappa, the intraclass correlation coefficient and mean absolute error, and the agreement was described as good to excellent.

Pairwise score differences between models were computed and shown in item-level heatmaps. Overall mean score differences fell within ±0.5, though some individual-item comparisons exceeded that range. The authors describe overall performance as similar across the models despite deviations on several questions.

The authors conclude the models could serve as auxiliary tools for clinical information retrieval and patient education, but only under professional supervision, with qualified clinicians checking all generated content before clinical use.

Sources

  1. Peer-reviewedInternational Journal of Gynecology & Obstetrics2026-10-17

The debate