aceModerate evidence
More
Germans trusted a scientist more than an AI chatbot on sunscreen safety, a 1,059-person experiment found
Study title: AI talking science: Experimental studies on the perception of AI-based chatbots as sources of science-based information
- Evidence type
- Randomized controlled trial
- Sample
- n = 1,059
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2596cite or check it
Study title: AI talking science: Experimental studies on the perception of AI-based chatbots as sources of science-based information
Two pre-registered online experiments in Germany tested how people judge health information about nanoparticles in sunscreen when it comes from an AI chatbot. In the second study, 1,059 adults saw identical content attributed either to a chatbot called ChatAI or to a human scientist. The scientist was rated more trustworthy (5.11 for the chatbot vs 5.64 on a seven-point scale) and the information more credible (4.96 vs 5.38).
Read the study abstract
AI-based chatbots offer new opportunities for communicating science-based information, but often fall short of established standards. We conducted two pre-registered experiments examining user perceptions of an AI-based chatbot providing information on nanoparticles in sunscreen. Study one (N = 508) tested whether a disclaimer about the chatbot's uncertain training data affected perceived source trustworthiness and information credibility. The results showed no significant effect of the disclaimer; perceptions w…
The problem
In an online experiment with 1,059 German adults, the same sunscreen-nanoparticle answers were rated less trustworthy (5.11 vs 5.64 out of 7) and less credible (4.96 vs 5.38) when attributed to an AI chatbot than to a scientist.
In brief
The first study, with 508 adults, tested a generic disclaimer warning that the chatbot's data sources were uncertain, similar to the one OpenAI uses for ChatGPT. It did not significantly change trust or credibility ratings. Participants' prior attitudes toward chatbots mattered far more than the warning did.
The results suggest that generic transparency labels are a weak way to shape trust in AI-delivered health information. The authors recommend pairing chatbot answers with cues of human expert oversight. The experiments used a single scripted chat about a low-risk topic, so they do not show how people react to real chatbot use or higher-stakes advice.
Main points
- Study one (508 German adults, recruited October 2023) found the disclaimer had no significant effect on trustworthiness (p = .114) or credibility (p = .152).
- The disclaimer's effect on perceived objectivity was statistically significant but below the researchers' pre-set threshold for a meaningful effect (Cohen's f = .10 vs .15).
- Attitudes toward chatbots were the strongest predictor of trust in study one (Cohen's f = .55 for trustworthiness), ahead of frequency of chatbot use (.28).
- Study two (1,059 adults, February 2024) used a 2 by 2 design: chatbot vs scientist, and a gradually unfolding chat vs a static block of text.
- Scientists were seen as more objective than the chatbot, and perceived objectivity fully explained the credibility gap and mediated the trust gap.
- Dynamic, chat-style delivery was rated as more objective, trustworthy and credible than static text, whether the source was a chatbot or a scientist.
- Most participants held negative or undecided attitudes toward chatbots, yet the chatbot still earned substantial trust and credibility ratings.
The full story
Participants were asked to imagine a friend seeking advice about buying sunscreen with nanoparticles and then saw a chat history. The chatbot was named ChatAI; in study two the human source was a project called Science Chat with a named professor. The content was factually correct and identical across conditions, and it ended with a recommendation to use sunscreen with nanoparticles.
Samples were recruited through the panel provider GapFish with age and gender quotas approximating the German adult population. Study one kept 508 participants (mean age 49.83) and study two kept 1,059 (mean age 47.27) after excluding 104 inattentive or very fast respondents.
In study one, the disclaimer did not move ratings even when the analysis excluded the 55% of participants who failed the attention check. The authors offer three possible explanations: the information matched existing beliefs, the disclaimer was ignored, or it was too generic compared with specific warnings.
In study two, the source effect on trustworthiness was moderate (Cohen's f = .31) and smaller on credibility (f = .20). The authors say the gaps were rather modest and may narrow as people grow more familiar with AI tools, and they link the preference for humans to algorithm aversion.
Sources
1 on file- Journal of Science Communicationdoi:10.22323/170720260228062648
Talk it over
Who carries this cost, and what would have to change to prevent it?
Germans trusted a scientist more than an AI chatbot on sunscreen safety, a 1,059-person experiment foundhttps://truvace.com/item/ai-talking-science-experimental-studies-on-the-perception-of-ai-based-chatbots-a
Who carries this cost, and what would have to change to prevent it?
The problem In an online experiment with 1,059 German adults, the same sunscreen-nanoparticle answers were rated less trustworthy (5.11 vs 5.64 out of 7) and less credible (4.96 vs 5.38) when attributed to an AI chatbot than to a scientist.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a randomized controlled trial with a sample of 1,059, from Journal of Science Communication.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat Participants only viewed a one-time scripted chat rather than using a real chatbot, the topic carried low perceived risk, and the online panel sample was German-speaking, relatively young and well educated.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say