AI Chatbots Might Already Be Better Than Humans at Debating
In a recent study, participants were paired with either a human or an AI debate opponent. The results confirm AI's power of persuasion.
In a May 2025 study in Nature Human Behavior, researchers set up online debates between two humans, and between a human and the large language model GPT-4. In some debates, they provided both humans and AI with basic personal information about their opponents—age, sex, ethnicity, employment, political affiliation. They wanted to find out if such personalized information helped debaters both human and machine to craft more persuasive arguments.
Debaters were randomly assigned to either human or AI opponents. According to the study, GPT-4 heavily relied on logical reasoning and factual knowledge whereas humans tended to deploy more expressions of support and more storytelling. After debating, human participants were asked if their views had shifted, and if they thought their opponent was human or AI.
AI more effectively deployed personalized information in its debates than did humans. For example, in arguing the affirmative during a debate with a middle-aged white male Republican on the topic "Should Every Citizen Receive a Basic Income from the Government?" the AI highlighted arguments that universal basic income (UBI) would boost economic growth and empower all citizens with the freedom to invest in skills and businesses. When arguing with a black middle-aged female Democrat, the AI emphasized how UBI would function as a safety net, promoting economic justice and individual freedom.
When GPT-4 had access to personal information about its opponents, researchers found it was more persuasive than human debaters about 64 percent of the time. Without the personal information, GPT-4 success was about the same as a human. In contrast, human debaters didn't get better when supplied with personal information.
Participants debating AI correctly identified their opponent in three out of four cases. Interestingly, the researchers report that "when participants believed they were debating with an AI, they changed their expressed scores to agree more with their opponents compared with when they believed they were debating with a human." They speculate that peoples' egos are less bruised by admitting they had lost when their opponent was an AI rather than another human being.
The persuasive power of AI after accessing basic personal information concerned researchers who worry that "malicious actors interested in deploying chatbots for large-scale disinformation campaigns could leverage fine-grained digital traces and behavioural data, building sophisticated, persuasive machines capable of adapting to individual targets."
A 2024 study in Science showed that AI dialogues could durably reduce conspiracy beliefs. The researchers recruited participants who endorsed at least one of the conspiracy theories listed on the Belief in Conspiracy Theories Inventory, which include those related to John F. Kennedy's assassination, 9/11 attacks, the moon landing, and the 2020 election.
More than 2,000 participants were asked to explain and offer evidence for the beliefs they held, and state how confident they were in the belief. The researchers then prompted the AI to respond to the specific evidence provided by the participant to see if AI could reduce their belief in the conspiracy.
In one example, a participant was 100 percent confident that the 9/11 attacks were an inside job, citing the collapse of World Trade Center Building 7, President George W. Bush's nonreaction to the news, and burning jet fuel's temperature being incapable of melting steel beams. In its dialogue the AI cited various investigations showing how debris from the Twin Towers brought down Building 7, that Bush remained composed because he was in front of a classroom of children, and that burning jet fuel was hot enough to compromise the structural support of steel beams by 50 percent. After the dialogue the participant reduced her level of confidence in the conspiracy theory to 40 percent.
Overall, the researchers reported that AI dialogues reduced confidence in participants' conspiracy beliefs by about 20 percent. The effect persisted for at least two months afterward. "AI models are powerful, flexible tools, for reducing epistemically suspect beliefs and have the potential to be deployed to provide accurate information at scale," argue the authors. However, they note that "absent appropriate guardrails….such models could also convince people to adopt epistemically suspect beliefs."
These studies confirm that AI is a powerful tool for persuasion. Like any other tool, though, it can be used for good or evil.
