Moderate evidence
More
AI graders were more consistent than pharmacy faculty in a study of 55 student essays
Study title: The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses
- Evidence type
- Controlled or validated comparison
- Sample
- n = 55
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2984cite or check it
Study title: The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses
A study in the American Journal of Pharmaceutical Education compared two AI models with pharmacy faculty on grading 55 deidentified drug information responses written by second-year pharmacy students. ChatGPT 5.2 reached an intraclass correlation of .90 and Gemini 2.5 Flash reached .72, against .39 for faculty.
Read the study abstract
Objective: To compare the reliability of two artificial intelligence (AI) models and pharmacy faculty in grading drug information (DI) responses. Methods: We retrospectively collected 55 deidentified DI responses written by second-year pharmacy students. A 32-point grading rubric was developed and five faculty graders completed two calibration sessions to ensure grading consistency. Each DI response was graded by three randomly assigned faculty and two AI models (OpenAI ChatGPT 5.2 and Google Gemini 2.5 Flash).…
The gain
In a study of 55 pharmacy student drug information responses, ChatGPT 5.2 (ICC .90) and Gemini 2.5 Flash (ICC .72) graded more consistently than faculty graders (ICC .39) using the same 32-point rubric.
In brief
Each response was graded by three randomly assigned faculty and by both AI models on a 32-point rubric. Faculty had completed two calibration sessions beforehand, yet their agreement with one another stayed lower than that of the AI models.
Median rubric scores were 25.9 points for faculty, 24.2 for ChatGPT and 24.7 for Gemini. Both AI models graded slightly harsher, by 1.2 to 1.7 points, about 5% of the rubric, and showed no proportional bias across essay quality.
Main points
- The study used 55 retrospectively collected, deidentified drug information responses from second-year pharmacy students.
- A 32-point rubric was developed, and five faculty graders completed two calibration sessions; each response was graded by three of them.
- The AI graders were OpenAI's ChatGPT 5.2 and Google's Gemini 2.5 Flash.
- Reliability was ICC .90 for ChatGPT, .72 for Gemini and .39 for faculty.
- Against the mean faculty score, both models had a mean absolute error of 2.1 points.
- Median scores did not differ between the two AI models.
- Neither model showed proportional bias, and grading differences were spread evenly regardless of essay quality.
The full story
Researchers built a 32-point rubric for drug information responses and had five faculty graders complete two calibration sessions to align their scoring. Each of the 55 student responses was then scored by three randomly assigned faculty and by both AI models.
Reliability was measured with intraclass correlation, and the mean faculty score served as a consensus benchmark for mean absolute error. Both ChatGPT and Gemini differed from that benchmark by 2.1 points on average.
The AI models scored lower than faculty: median 24.2 for ChatGPT and 24.7 for Gemini versus 25.9 for faculty. The authors describe this as about a 5% difference on the rubric, and Spearman's correlation found no proportional bias.
Sources
1 on file- American Journal of Pharmaceutical Educationdoi:10.1016/j.ajpe.2026.102124
Talk it over
ChatGPT 5.2 reached an ICC of .90 and Gemini 2.5 Flash .72, while faculty reached .39. The study never checked the scores against a known correct grade, so what would agreement among graders need to be paired with to count as good grading?
AI graders were more consistent than pharmacy faculty in a study of 55 student essayshttps://truvace.com/item/the-use-of-artificial-intelligence-and-its-reliability-in-grading-drug-informati
ChatGPT 5.2 reached an ICC of .90 and Gemini 2.5 Flash .72, while faculty reached .39. The study never checked the scores against a known correct grade, so what would agreement among graders need to be paired with to count as good grading?
Answer in the threadBoth AI models graded 1.2 to 1.7 points harsher than faculty on the 32-point rubric. How would you weigh that gap against their higher agreement with each other?
Answer in the threadIf you were one of the second-year pharmacy students whose drug information responses were graded, which would you want deciding your score: three faculty, an AI model or a mix, and why?
Answer in the thread
If you have more time
The study used 55 deidentified responses from second-year pharmacy students and one rubric. What other kinds of student work or rubrics would you want tested before anyone relied on AI graders?
Answer in the threadFaculty agreement stayed at .39 despite two calibration sessions. What would you look up or ask the five faculty graders to understand why they disagreed?
Answer in the thread
ace
What readers say