52/100

Moderate evidence

Cite
More
EmailLinkedInX

AI sentiment tools struggle to match experts on pregnancy weight posts, a Reddit study finds

Study title: Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study

Evidence type
Benchmark or lab evaluation
Sample
n = 150
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study

Researchers tested how well 10 sentiment models classify the tone of Reddit discussions about weight gain in pregnancy, using 150 held-out posts labelled by expert consensus. Gemini-3-Flash scored highest with a weighted F1 of 0.662, followed by GPT-OSS at 0.645, and confidence intervals across the leading models overlapped.

Read the study abstract

Background Pregnancy-related weight gain is both physiologically complex and socially charged, and online discussions often combine clinical information with implicit emotional distress. Although automated sentiment analysis could support research on pregnancy-related discourse, model performance in this context remains inadequately characterized relative to expert human judgment. Objective This study aimed to benchmark contemporary sentiment classification models on pregnancy-related weight discussions, with em…

Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study
"Prélèvement sérologie au Laboratoire Hopital Douala Cameroun 10" by Happi Raphael, CC BY-SA 4.0.

The problem

In a JMIR Formative Research benchmark of 10 models on 150 Reddit pregnancy-weight posts, the best (Gemini-3-Flash) reached a weighted F1 of 0.662 against expert consensus, and the authors say such tools cannot stand alone as a measure of patient experience.

In brief

The human experts themselves found the task hard: the two primary experts agreed on only 59.0% of posts before adjudication (Cohen κ=0.344). Among Gemini-3-Flash's errors, 55.8% (29 of 52) fell on posts where those experts had disagreed, so much of the model error reflects real ambiguity in the text.

The authors conclude that broad polarity classification can help prioritise posts and generate hypotheses but is not enough by itself to measure how pregnant people experience their care or bodies. Agreement between the two AI models was high on posts (92.3%) but low on comments (46.3%).

Main points

  1. The team built a corpus of 15,619 Reddit posts and comments through LLM-guided keyword expansion and set up an adjudicated reference set of 200 posts.
  2. On the 150-post held-out test, Gemini-3-Flash scored a weighted F1 of 0.662 (κ=0.478) and GPT-OSS 0.645 (κ=0.436); BERTweet scored 0.623 and RoBERTa 0.605.
  3. With fine-tuning in 5-fold cross-validation, RoBERTa's mean F1 rose to 0.708 (SD 0.087) while BERTweet's was 0.616 (SD 0.079).
  4. In a separate 106-post extension, expert agreement was 64.2% (κ=0.367); Gemini-3-Flash and GPT-OSS scored 0.633 and 0.680, and both identified 56 of 58 expert-negative posts.
  5. Agreement between Gemini-3-Flash and GPT-OSS was 92.3% on 400 posts (κ=0.797) but only 46.3% on 1,500 comments (κ=0.102).
  6. All 714 post IDs retrieved through the API could be independently recovered through Arctic Shift.

The full story

The study started with a pool of 15,619 Reddit posts and comments about pregnancy-related weight, gathered with a keyword strategy expanded by a large language model. Two primary experts labelled sentiment and adjudicated a 200-post reference set, of which 150 posts formed a fixed test set. Models were scored with weighted metrics and 1,000-resample bootstrap confidence intervals.

No architecture came out uniformly ahead. The two general-purpose LLMs led under direct use, but a fine-tuned RoBERTa reached a mean F1 of 0.708 in cross-validation. The authors say performance depended on how the model was adapted, how ambiguous the labels were, and whether the text was a post or a comment.

When applied to unlabeled data, Gemini-3-Flash and GPT-OSS agreed closely on 400 posts but diverged on 1,500 comments, suggesting short replies are much harder to classify. In an exploratory analysis, both models caught 56 of 58 posts the experts rated negative, which the authors frame as a way to prioritise posts for human review.

Sources

1 on file
  1. Peer-reviewed2026-11-13

Talk it over

Three questions to start on, two more if you have time
The two primary experts agreed on only 59.0% of posts before adjudication, and 29 of Gemini-3-Flash's 52 errors fell on posts where they disagreed. What does it mean for a model to be wrong on such posts?

Start with a question

AI sentiment tools struggle to match experts on pregnancy weight posts, a Reddit study findshttps://truvace.com/item/benchmarking-large-language-models-for-sentiment-analysis-in-pregnancy-related-w

  1. The two primary experts agreed on only 59.0% of posts before adjudication, and 29 of Gemini-3-Flash's 52 errors fell on posts where they disagreed. What does it mean for a model to be wrong on such posts?

    Answer in the thread
  2. Both models caught 56 of 58 posts the experts rated negative. How would you weigh that use for prioritising posts against the best weighted F1 of 0.662?

    Answer in the thread
  3. Agreement between the two models was 92.3% on posts but 46.3% on comments. Who should decide which kinds of text these tools are used on, and what should they check first?

    Answer in the thread
If you have more time
  • The posts came from Reddit. Which pregnant people might not post there, and how could their views change what a sentiment tool finds?

    Answer in the thread
  • A fine-tuned RoBERTa reached a mean F1 of 0.708 in cross-validation. What evidence would you want to see before choosing it over a general-purpose model?

    Answer in the thread

What readers say