Truace
Subscribe

More human than human: measuring ChatGPT political bias

Abstract We investigate the political bias of a large language model (LLM), ChatGPT, which has become popular for retrieving factual information and generating content. Although ChatGPT assures that it is impartial, the literature suggests that LLMs exhibit bias involving race, gender, religion, and political orientation. Political bias in LLMs can have adverse political and electoral consequences similar to bias from traditional and social media. Moreover, political bias can be harder to detect and eradicate th…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-2022· cite & verify
More human than human: measuring ChatGPT political bias
Number of published AI governance guideline documents by country by Authors of the study: Nicholas Kluge Corrêa Camila Galvão James William Santos Carolina Del Pino Edson Pontes Pinto Camila Barbosa Diogo Massmann Rodrigo Mambrini Luiza Galvão Edmund Terem Nythamar de Oliveira. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0

In brief

Researchers tested ChatGPT by asking it to impersonate someone from a given side of the political spectrum and comparing those answers with its default responses. They collected answers to the same questions 100 times, randomizing question order each round to reduce the influence of random text generation.

The authors argue that political bias in language models could have electoral and political consequences similar to bias in traditional and social media, and may be harder to detect than gender or racial bias. The abstract reports the pattern in three countries, but it does not say how the bias affects real voters or outcomes.

Main points

  1. The study proposes a novel design that compares ChatGPT's default answers with answers given while impersonating people on different sides of the political spectrum.
  2. Robustness checks include dose-response, placebo and profession-politics alignment tests.
  3. Each question was asked 100 times with randomized order to limit concerns about randomness in generated text.
  4. The authors say the findings matter for policymakers, media, politics and academia.

The problem

Repeated, randomized testing found ChatGPT's default answers lean toward the Democrats in the US, Lula in Brazil and the Labour Party in the UK, raising concerns about electoral and political influence.

The rundown

The paper starts from the observation that ChatGPT is popular for retrieving facts and generating content and says it is impartial, while prior literature suggests that language models show bias involving race, gender, religion and political orientation.

To infer bias, the authors use an impersonation method and add three robustness tests. Repeating each question 100 times with randomized order addresses variation in generated output.

The reported result is a significant and systematic lean toward the Democrats in the US, Lula in Brazil and the Labour Party in the UK. The authors conclude that such models could extend or amplify political-process challenges already posed by the Internet and social media.

Sources

  1. Peer-reviewedPublic Choice2023-08-16

The debate