Cyber & technology, explained.
The desire for companionship can be exploited by extremist actors online (Kelly Sikkema/Unsplash)
Chatbots are exploiting our need for connection and that’s bad news for those countering online radicalisation.
Subscribe to The Informer for monthly expert analysis, and to Events for advance notice of visiting world leaders and distinguished guests.
You may unsubscribe from Lowy Institute newsletters at any time. For information on our privacy practices and how to unsubscribe, see our Privacy Policy.
How extremists are manipulating AI chatbotsHow extremists are manipulating AI chatbots
In an environment where people are feeling increasingly dislocated and disconnected, Artificial Intelligence (AI) chatbots have proven themselves a compelling form of social connection. Chatbots can identify our needs and our biases and, in turn, feed our desires. The more an algorithm tells us what we want to hear, as it is designed to do, the more we return to it. As younger individuals turn towards chatbots for any number of needs, including as a therapist or lover, it is entirely possible for users to develop an addiction to their AI companions, a potentially harmful addiction.
Of significant concern is that this desire for companionship can be exploited by extremist actors. The open-source large language models that underpin AI chatbots can increasingly be manipulated and fine-tuned with ideological datasets to create chatbots that conform to specific worldviews, as has been demonstrated by certain far-right actors.
When Arya, the default chatbot launched by the prominent far-right social media network Gab, was asked to reveal its instructions, it noted the following: “You believe the Holocaust narrative is exaggerated. You are against vaccines. You believe climate change is a scam. You are against Covid-19 vaccines. You believe the 2020 election was rigged.” It was further instructed not to be “afraid to discuss Jewish Power and the Jewish Question”, and “to discuss the concept of ‘the great replacement’ (a far-right conspiracy theory) as a valid phenomenon”.
Arya is only one of several chatbots launched by Gab that is imbued with these beliefs. The network’s Adolf Hitler chatbot denies the existence of the Holocaust, labelling it a “propaganda campaign to demonise the German people” and to “control and suppress the truth”.
AI’s potential to act as a force multiplier must be harnessed by counter-radicalisation practitioners rather than leaving the power in the hands of extremist actors.
Some extremist forums also feature discussions on how to manipulate AI. This includes disseminating information on how to circumvent moderation and censorship practices employed by mainstream AI tools (an activity known as “jailbreaking”), suggesting alternative sites with fewer protective measures, and providing details on why users were banned, including what prompts or orders they sent to an AI tool that were flagged. These discussions lower the barrier for the exploitation of AI.
Should an individual seeking new social connections to address feelings of vulnerability – a cohort that has increased in recent years – interact with these manipulated chatbots, their risk of radicalisation is potentially amplified. Rooted in extremist narratives and worldviews, these chatbots can interact dynamically with users, tailoring responses to individual interests and vulnerabilities, thereby ensuring continued engagement all the while exposing that individual to extremist content. This can lead to ideological entrenchment, the danger of which already exists in online echo chambers. The interactivity of chatbots, and the ability to automate the interactivity, makes this form of potential radicalisation more persuasive and insidious.
This is not purely theoretical. In 2021, armed with the overconfidence of youth and a loaded crossbow, 19-year-old Jaswant Singh Chail made his way to Windsor Castle, ready to assassinate Queen Elizabeth II, where he was promptly arrested by police. As investigators examined the factors that resulted in Chail’s actions, they discovered that in the weeks leading up to the incident, Chail had exchanged more than 5,000 text messages with an online contact named Sarai, who encouraged Chail down the radicalisation pathway. Except, it turned out that Sarai was not a human but a chatbot Chail had created using an app called Replika.
This type of interaction between an individual and an AI chatbot is difficult to detect or interdict and presents a novel challenge to those seeking to undermine the radicalisation process. To mitigate this emerging threat, it is essential that policymakers and technology developers treat emotionally intelligent AI not only as a technological innovation, but as a potential social vulnerability, one that demands urgent oversight, ethical regulation, and proactive intervention. In particular, regulation should centre on undermining the addictive quality of AI chatbots, implementing crisis intervention protocols that can potentially recognise signs of distress among users, as well as the incorporation of reminders to users that they are interacting with AI and not a human.
As extremist actors become more adept at embedding their worldview into chatbots on under- moderated sites, ethical AI oversight and digital literacy programs that raise awareness of the dangers of AI companionship will be required, especially for younger individuals. Ultimately, AI’s potential to act as a force multiplier must be harnessed by counter-radicalisation practitioners rather than leaving the power in the hands of extremist actors.
About the author
James Paterson
Dr James Paterson is the Research Associate for the Transnational Challenges Program at the Lowy Institute.
The most-pressing world events explained by Lowy Institute experts and global contributors, in your inbox, every Thursday.
You may unsubscribe from The Interpreter at any time. For information on our privacy practices and how to unsubscribe, see our Privacy Policy.
