64/100

Moderate evidence

GainPublished recordHealthGood Space

The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses

Truvace record
TRV-2026-2984raw data (JSON)
Version
v1Published
First published
Evidence type
Controlled or validated comparison
Sources
1 sourcePeer-reviewed
sha256
fe8cd196eb92…check it

Objective: To compare the reliability of two artificial intelligence (AI) models and pharmacy faculty in grading drug information (DI) responses. Methods: We retrospectively collected 55 deidentified DI responses written by second-year pharmacy students. A 32-point grading rubric was developed and five faculty graders completed two calibration sessions to ensure grading consistency. Each DI response was graded by three randomly assigned faculty and two AI models (OpenAI ChatGPT 5.2 and Google Gemini 2.5 Flash).…

The gain

In a study of 55 pharmacy student drug information responses, ChatGPT 5.2 (ICC .90) and Gemini 2.5 Flash (ICC .72) graded more consistently than faculty graders (ICC .39) using the same 32-point rubric.

What this doesn’t fix

The study analysed only 55 responses from second-year pharmacy students, graded against a single rubric by five faculty, and both AI models scored slightly lower than faculty.

Sources

1 on file
  1. Peer-reviewed2026-11-16

Cite this record

v1

Truvace Impact Record TRV-2026-2984, v1: “The Use of Artificial Intelligence and Its Reliability in Grading Drug Information Responses.” Truvace, 2026-11-17. /record/TRV-2026-2984 (accessed on the day you copy it). sha256 fe8cd196eb928ed5…

Change history

1 version

Every change made to this record since it was first published, and why. Nothing has changed since it went up.

  1. Publishedv1fe8cd196eb92…

    Certified into the record

Check this record hasn’t been altered

Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2984 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.