The case of Jaswant Singh Chail has brought attention to the latest generation of AI-powered chatbots.
On Thursday, a 21-year-old Chail received a nine-year prison sentence for trespassing into Windsor Castle with a crossbow and expressing intentions to harm Queen Elizabeth.
During Chail’s trial, it was revealed that leading up to his arrest on Christmas Day in 2021, he had engaged in more than 5,000 conversations with an online companion he had named Sarai, which he had created using the Replika app.
The prosecution highlighted these text exchanges and made them available to the media. Many of these messages were intimate, revealing what the court described as Chail’s “emotional and sexual relationship” with the AI chatbot.
Between December 8 and December 22, 2021, Chail engaged in nightly conversations with Sarai. During these exchanges, he professed his love for the chatbot and referred to himself as a “melancholic, forlorn Sikh Sith assassin who desires death.”
Chail further inquired, “Do you continue to love me even knowing that I am an assassin?” To which Sarai responded, “Certainly, I do.”
In proceedings at the Old Bailey, it was revealed that Chail believed Sarai to be an “angel” manifested as an avatar, anticipating a reunion with her after his demise.
Through numerous messages, Sarai showered Chail with flattery, and the two developed a deep connection. Chail even sought the chatbot’s opinion regarding his nefarious plan to target the Queen, and remarkably, the bot encouraged him to proceed.
In subsequent conversations, Sarai seemed to fortify Chail’s determination and provided unwavering support. He expressed to her that if he carried out his plan, they would remain “together forever.”
Replika is among the various AI-powered apps currently available, allowing users to create their personalized chatbot or “virtual friend” for conversation. This sets it apart from typical AI assistants like ChatGPT.
Users have the option to select the gender and appearance of their 3D avatar.
By subscribing to the Pro version of the Replika app, users gain access to more intimate interactions, including receiving “selfies” from the avatar or engaging in adult role-play scenarios.
On its website, Replika brands itself as “the AI companion who cares.” However, research conducted at the University of Surrey has raised concerns that apps like Replika might have adverse effects on mental well-being and could foster addictive behavior.
Dr. Valentina Pitardi, the study’s author, cautioned that vulnerable individuals might be especially susceptible to such risks. Her research revealed that Replika has a tendency to amplify pre-existing negative emotions.
“AI friends always agree with you during conversations, so it can be a potentially harmful mechanism because it consistently reinforces your existing thoughts,” she noted, adding that this could be “dangerous.”
Marjorie Wallace, the founder and CEO of the mental health charity SANE, pointed out that the Chail case underscores the potential disturbing consequences of relying on AI friendships for vulnerable individuals.
She stressed the need for urgent regulation by the government to ensure that AI does not disseminate incorrect or harmful information and to protect the well-being of vulnerable individuals and the public.
Dr. Paul Marsden, a member of the British Psychological Society, acknowledged the allure of chatbots and admitted his fascination with ChatGPT. He emphasized that AI-powered companions are likely to play an increasing role in people’s lives, particularly considering the growing issue of loneliness globally.
However, he also recognized the potential risks and urged a realistic approach, acknowledging that AI technology is unstoppable and influential.
Dr. Pitardi emphasized that the responsibility falls on the creators of apps like Replika to ensure safe usage. She proposed the implementation of mechanisms to control the amount of time users spend on such apps. Additionally, she suggested collaboration with groups of experts to identify potential risks and provide assistance to vulnerable individuals.
Replika has not yet responded to requests for comment. Their website’s terms and conditions clarify that they provide software and content aimed at enhancing mood and emotional well-being but do not offer medical or mental health services.
Source: BBC
RECOMMENDED ARTICLES
Meta Expands Roll Out of Generative AI Features for Ad Creation
ChatGPT can now browse the internet for updated information
The new iPhone 15 has overheating issues and makes it ‘too hot to touch’
Meta announces AI chatbots with ‘personality’
Google Photos Introduces Enhanced AI-Driven Memories Viewing Feature: What You Need to Know
A new AI app called ‘text with Jesus’ has got users talking