64/100

Moderate evidence

Cite
More
EmailLinkedInX

AI graders were more consistent than pharmacy faculty in a study of 55 student essays

Study title: The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses

Evidence type
Controlled or validated comparison
Sample
n = 55
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses

A study in the American Journal of Pharmaceutical Education compared two AI models with pharmacy faculty on grading 55 deidentified drug information responses written by second-year pharmacy students. ChatGPT 5.2 reached an intraclass correlation of .90 and Gemini 2.5 Flash reached .72, against .39 for faculty.

Read the study abstract

Objective: To compare the reliability of two artificial intelligence (AI) models and pharmacy faculty in grading drug information (DI) responses. Methods: We retrospectively collected 55 deidentified DI responses written by second-year pharmacy students. A 32-point grading rubric was developed and five faculty graders completed two calibration sessions to ensure grading consistency. Each DI response was graded by three randomly assigned faculty and two AI models (OpenAI ChatGPT 5.2 and Google Gemini 2.5 Flash).…

The gain

In a study of 55 pharmacy student drug information responses, ChatGPT 5.2 (ICC .90) and Gemini 2.5 Flash (ICC .72) graded more consistently than faculty graders (ICC .39) using the same 32-point rubric.

In brief

Each response was graded by three randomly assigned faculty and by both AI models on a 32-point rubric. Faculty had completed two calibration sessions beforehand, yet their agreement with one another stayed lower than that of the AI models.

Median rubric scores were 25.9 points for faculty, 24.2 for ChatGPT and 24.7 for Gemini. Both AI models graded slightly harsher, by 1.2 to 1.7 points, about 5% of the rubric, and showed no proportional bias across essay quality.

Main points

  1. The study used 55 retrospectively collected, deidentified drug information responses from second-year pharmacy students.
  2. A 32-point rubric was developed, and five faculty graders completed two calibration sessions; each response was graded by three of them.
  3. The AI graders were OpenAI's ChatGPT 5.2 and Google's Gemini 2.5 Flash.
  4. Reliability was ICC .90 for ChatGPT, .72 for Gemini and .39 for faculty.
  5. Against the mean faculty score, both models had a mean absolute error of 2.1 points.
  6. Median scores did not differ between the two AI models.
  7. Neither model showed proportional bias, and grading differences were spread evenly regardless of essay quality.

The full story

Researchers built a 32-point rubric for drug information responses and had five faculty graders complete two calibration sessions to align their scoring. Each of the 55 student responses was then scored by three randomly assigned faculty and by both AI models.

Reliability was measured with intraclass correlation, and the mean faculty score served as a consensus benchmark for mean absolute error. Both ChatGPT and Gemini differed from that benchmark by 2.1 points on average.

The AI models scored lower than faculty: median 24.2 for ChatGPT and 24.7 for Gemini versus 25.9 for faculty. The authors describe this as about a 5% difference on the rubric, and Spearman's correlation found no proportional bias.

Sources

1 on file
  1. Peer-reviewed2026-11-16

Talk it over

Three questions to start on, two more if you have time
ChatGPT 5.2 reached an ICC of .90 and Gemini 2.5 Flash .72, while faculty reached .39. The study never checked the scores against a known correct grade, so what would agreement among graders need to be paired with to count as good grading?

Start with a question

AI graders were more consistent than pharmacy faculty in a study of 55 student essayshttps://truvace.com/item/the-use-of-artificial-intelligence-and-its-reliability-in-grading-drug-informati

  1. ChatGPT 5.2 reached an ICC of .90 and Gemini 2.5 Flash .72, while faculty reached .39. The study never checked the scores against a known correct grade, so what would agreement among graders need to be paired with to count as good grading?

    Answer in the thread
  2. Both AI models graded 1.2 to 1.7 points harsher than faculty on the 32-point rubric. How would you weigh that gap against their higher agreement with each other?

    Answer in the thread
  3. If you were one of the second-year pharmacy students whose drug information responses were graded, which would you want deciding your score: three faculty, an AI model or a mix, and why?

    Answer in the thread
If you have more time
  • The study used 55 deidentified responses from second-year pharmacy students and one rubric. What other kinds of student work or rubrics would you want tested before anyone relied on AI graders?

    Answer in the thread
  • Faculty agreement stayed at .39 despite two calibration sessions. What would you look up or ask the five faculty graders to understand why they disagreed?

    Answer in the thread

What readers say