LLM versus human thematic analysis of interviews with Ugandan women after fistula surgery

Source article: Efficiency vs Depth in AI-Generated and Human-Synthesized Thematic Analyses of Ugandan Women's Experiences of Obstetric Fistula: Comparative Qualitative Study

Background Limited recent literature has evaluated the use of large language models (LLMs) in the qualitative analysis of health data; more research is needed to expand the generalizability of LLM use and to evaluate potential ethical considerations. Objective Our research sought to (1) describe the process of using AI to analyze qualitative in-depth interview data, (2) identify similarities and differences between the human and AI-generated analyses to compare the quality and rigor of the two techniques and des…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1653· cite & verify

Trace impact readingContested

P 67

Both sides are scored from claims and sources, not community votes.

G 71

In brief

Researchers compared human and AI thematic analysis of hour-long interviews with 17 Ugandan women recovering from fistula surgery. The AI was Versa, a UCSF-developed tool powered by ChatGPT-4o, run inductively and deductively. The two approaches overlapped substantially, but the human team used a larger codebook and produced richer summaries.

The result matters for health research teams weighing AI to speed up qualitative work on sensitive patient experiences. The AI saves considerable time but still depends on people to write prompts, enter data and safeguard interpretation and ethics. The findings come from one small sample with one tool, so how far they generalize remains open.

Main points

Seventeen Ugandan women recovering from fistula surgery took part in semistructured interviews of about an hour each.
Inductively, Versa identified 39 codes against 54 used by the human researchers.
Versa's analyses were generally accurate in representing the data, but the human analysis was more robust.
The human analysis took roughly 15 hours versus about 2.5 hours for Versa.
The authors conclude significant human engagement is needed to maintain ethical and interpretive rigor.

The gain

Using the Versa LLM to analyze interviews with women recovering from fistula surgery cut analysis time from about 15 hours to about 2.5 hours.

The problem

Versa's analysis lacked the narrative depth, compelling excerpts and range of the human synthesis, so it could not capture lived experience without substantial human involvement.

The rundown

The interviews explored the women's mental, physical and overall health trajectories. After translation and transcription, humans and Versa each performed thematic analysis, and the AI was run with both inductive and deductive strategies.

Outputs were compared on code frequency and alignment, thematic robustness, analysis quality including narrative depth and excerpt accuracy, and efficiency measured in person-hours.

The human synthesis stood out for including compelling excerpts and broader summary content. Versa still relied on human researchers to build prompts and input all the data into the chat, which limits how automated the workflow is.

What this doesn’t fix

The study is a small comparison nested within a larger mixed methods study, and the authors note that more research is needed to expand the generalizability of LLM use and to evaluate ethical considerations.