ChatGPT matched dental experts on review quality in just 52 of 100 cases, study finds

The chatbot was harsher than the experts: it classed 85 reviews as critically low confidence, compared with 47 by human consensus. When the rules for handling 'Partial Yes' ratings were changed, direct agreement rose to 83%, but the chance-corrected kappa stayed similar at 0.173.

The authors concluded the findings do not support using ChatGPT-5.5 as an independent AMSTAR 2 assessor. Any exploratory use would need item-by-item checks by reviewers with methodological expertise.

The team used AMSTAR 2, a tool for appraising systematic reviews, on 100 endodontology reviews with meta-analysis. Three human experts produced a consensus rating, and ChatGPT-5.5 generated item-level ratings and overall confidence classifications through three newly created accounts.

Agreement was assessed using direct agreement and linearly weighted Cohen's kappa. Under the primary criterion (Partial Yes = Yes), overall confidence matched in 52% of reviews. Under the alternative criterion (Partial Yes = No), agreement rose to 83%, while kappa stayed similar at 0.173, so the higher figure reflected how Partial Yes was handled rather than better chance-corrected agreement.

Agreement varied substantially across individual AMSTAR 2 items. The chatbot's outputs were frequently consistent across the three accounts, but that consistency did not ensure concordance with human judgement. The authors say any exploratory use requires item-by-item verification by reviewers with methodological expertise.