± SCIENCE Today’s artificial intelligence still faces two major challenges. One is that, in most industries, data exist…+ SCIENCE Deep learning takes on protein folding In 1972, Anfinsen won a Nobel prize for demonstrating a connection bet…± SCIENCE Significance Deep neural networks are currently the most successful machine-learning technique for solving a…+ SCIENCE Deep learning allows computational models that are composed of multiple processing layers to learn representa…+ HEALTH For the task of analyzing survival data to derive risk factors associated with mortality, physicians, researc…+ SCIENCE Artificial intelligence (AI) coupled with promising machine learning (ML) techniques well known from computer…+ SCIENCE Deep learning is revolutionizing many areas of science and technology, especially image, text, and speech rec…+ SCIENCE Current deep-learning models are mostly built upon neural networks, i.e. multiple layers of parameterized dif…
TruaceTracing the truth around AIMonday, July 13, 2026
Subscribe
Crime·The Trace·Automated dual reading·Published 2026-07-13

generative AI impact on scale and handling of harmful content in Trust & Safety

Source article: How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

Generative AI (GenAI) is a powerful technology poised to reshape Trust & Safety. While misuse by attackers is a growing concern, its defensive capacity remains underexplored. This paper examines these effects through a qualitative study with 43 Trust & Safety experts across five domains: child safety, election integrity, hate and harassment, scams, and violent extremism. Our findings characterize a landscape in which GenAI empowers both attackers and defenders. GenAI dramatically increases the scale and speed of…

TRV-2026-0197Peer-reviewedPermanent record — cite & verify
Trace impact reading

Contested: both sides are scored from claims and sources, not community votes.

P 68The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.
How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape

Safety inspections at Yaquina Head (36579289703) by BLM Oregon & Washington. Public domain

The quick read

On April 13, 2026, a peer-reviewed CHI paper reported a qualitative study of 43 Trust & Safety experts across child safety, election integrity, hate and harassment, scams, and violent extremism. It found generative AI both expands attacker capabilities and offers new defensive tools for detection and mitigation.

The dual-use framing matters because it moves beyond misuse-only narratives to show how the same capability that lowers barriers to propaganda and deepfakes could also enable large-scale detection, investigations, and moderator support. Uncertainty remains because defensive benefits are largely envisioned rather than demonstrated in operational deployment.

Main points
  • Qualitative study involved 43 Trust & Safety experts across five domains.
  • Five domains examined were child safety, election integrity, hate and harassment, scams, and violent extremism.
  • Defensive uses envisioned include conducting investigations, deploying persuasive counternarratives, improving moderator wellbeing, and offering user support.
Gain

Trust and Safety defenders can use generative AI to detect and mitigate harmful content at scale and support investigations and moderator wellbeing.

Problem

Generative AI increases the scale and speed of Trust and Safety attacks and lowers barriers to creating sophisticated propaganda and deepfakes.

The rundown

The paper reports a qualitative study with 43 experts spanning child safety, election integrity, hate and harassment, scams, and violent extremism to characterize how generative AI reshapes attack and defense.

Attacker empowerment is described as increased scale and speed and lower barriers to harmful content including sophisticated propaganda and deepfakes, while defender empowerment includes detection at scale, investigations, counternarratives, moderator wellbeing, and user support.

What this doesn’t fix

Findings are based on a qualitative study of 43 experts and include envisioned defensive uses rather than measured deployments at scale.

Sources

Reader signal

How should this claim be treated?

The debate