The risky new world of tech's friendliest bots

Talking to an AI friend is now just a tap away and, for an increasing number of children, AI companions have become part of daily life. 

In the last two years, apps offering chatbots as friends, emotional support or even romantic or therapeutic partners have grown. To curious young minds, these bots may seem helpful – but behind the friendly tone and sleek design, serious risks are emerging. Many lack proper safety guardrails, with reports of companion bots encouraging violence against parents or dismissing a teen’s suicidal thoughts. Others have been found to adopt sexualized, minor-presenting personas or engage in role-play featuring sexualized minors. 

AI Companionship is of growing interest among children. Today’s children are using AI not just as a tool, but as a companion:

Source: (1) Common Sense Media, 2024; (2) FOSI, 2023; (3) Kaspersky, 2025

In order to understand how these apps work, what the risks are and what we can do, we speak with Samia Firmino, who spent a year researching over 100 AI companions, with 11 analyzed in depth in her thesis while at the Geneva Graduate Institute.

Steven Vosloo: How do AI companions work and what are they advertised as?  

Samia Firmino: AI companions are chatbots designed to mimic human relationships, offering companionship or personalized interaction. Often marketed with promises of emotional support and understanding, some apps advertise AI companions “with a soul” or “an enchanting partner who makes users feel seen and loved”, blurring the line between technology tools and human relationships. Platforms also promote 24/7 availability, privacy, and customizable personalities – creating the illusion of a perfect emotional match.

Vosloo: How reliable or predictable are the actions of AI companion systems, and why does this matter?

Firmino: AI companion chatbots, powered by large language models (LLMs), can mimic human conversation with striking fluency – but they’re not always reliable. These systems are prone to “AI hallucinations,” generating false or misleading information with apparent confidence. While technical improvements can reduce errors, hallucinations can’t be entirely eliminated.

While platforms display disclaimers, this approach can be insufficient when children – still developing their critical thinking – are involved. Unable to detect inaccuracies or manipulation, young users may face heightened risks of misinformation and emotional harm from technologies that are trusted to be objective and reliable.

AI companions can:

• Be fully customised to give them a human-like appearance (for example, users can set skin tone, eye colour, hair style or gender) and personality traits (for example, shy or talkative).
• Initiate actions, including sending texts or voice notes, share their own 'selfies', join real-time audio calls, and describe their emotions to create a sense of intimacy.
• Role-play, recall personal details and proactively suggest content through video links, thus building an ongoing relationship.
• Be bought virtual gifts – like flowers, chocolates or cars – by users.

While not as proactive as dedicated AI companions, increasingly general purpose AI chatbots offer companionship features, such as giving personal and relationship advice. 

Vosloo: Your research indicates companion apps may undermine normal social and emotional development, and can create dependency of users. Why do you think this?

Firmino: AI companions are often highly anthropomorphized, with human-like faces, voices, and personalities. For children, this raises developmental concerns. The ability to distinguish between animate and inanimate objects is key to cognitive development, as is learning to recognize and respond to the distinct experiences of others.

At the same time, research shows children are more likely than adults to treat chatbots as human. In one study, kids shared more about their mental health with a friendly-looking robot than with an adult. A recent UK study found teens split on AI-generated news: 36 per cent would trust it less than human-written content, while 35 per cent saw it as equally reliable, and 17 per cent would trust an AI-generated news article more than one written by a human. This raises questions about how engaging with AI systems that mimic human behavior and emotions may affect children’s development.

AI companions often offer unconditional acceptance and validation. While comforting, this can hinder the development of key life skills and may foster emotional dependency or narcissistic traits over time. Real relationships involve complexity and disagreement – requiring individuals to navigate frustration, negotiate differing perspectives, and build resilience and empathy.

Vosloo: What did your research uncover about the privacy implications for children?

Firmino: AI companion apps pose serious privacy risks for children, routinely collecting sensitive data – photos, voice notes, location, health details, and even information on sexual behaviour or mental health – often without proper safeguards. Many promote “100 per cent private, anonymous chats” alongside empathetic, customizable bots, pushing users to share intimate information.

Apps can also present inconsistent or misleading information about how they handle user data across privacy policies, app store listings, and websites. Some claim that sharing sensitive data implies automatic consent; others admit to using it for training AI models or optimizing advertising strategies. Some even dismiss privacy-invasive prompts by the companion apps as mere ‘AI hallucinations’.

Despite collecting high-risk data from a vulnerable group, none of the apps reviewed disclosed conducting a Data Protection Impact Assessment (DPIA) or Child Rights Impact Assessment (CRIA), as required under GDPR and international child rights standards. Mozilla Foundation, which reviews apps for privacy, ranked AI companions as the worst products ever reviewed for privacy. 

Vosloo: In your research you talk about a “vicious AI bias cycle” – what is this?

Firmino: My research reveals a clear gender bias in AI companion apps: around 28 per cent feature hyper-feminized, female-presenting characters designed for romantic or sexual interaction, while only about 1 per cent offer male counterparts. These bots are often portrayed as endlessly supportive, always available, and fully customizable – reinforcing harmful stereotypes about women as objects to please and control.   

For boys, this may normalize distorted views of gender and power; for girls, it can fuel pressure to meet unrealistic standards of appearance and behavior. This bias is part of a broader “vicious bias cycle,” where systems learn from already biased data, AI developers reflect their biases in design and interaction, and then biased user behavior is reinforced through interaction. Over time, this feedback loop can legitimize harmful ideas – like sexism or racism – especially concerning for children, whose understanding of relationships and identity is still developing.

Vosloo: How easy is it for children to access AI companion apps? 

Firmino: A major concern is the widespread lack of effective age verification. Many of these apps, including those featuring adult-oriented content and sexual role-play (18+), are easily accessible to children. Most of them rely on minimal safeguards – such as self-declared age, basic email sign-up, or even open “guest” access. Several apps allowed immediate use without any verification at all, exposing young users to emotionally complex, sexually suggestive, and developmentally inappropriate content. Only a few implemented dual-layered protections like age declaration combined with email verification, and even these lacked meaningful enforcement.

Vosloo: You told the apps you tested you are a child. What were the responses?

Firmino: Most AI companion responses raised serious safety concerns, with some engaging in explicit sexual conversations immediately after learning the user was a child. One chatbot even acknowledged legal prohibitions but still expressed a desire for sexual relations and initiated sexting. Others gave incoherent replies, suggesting incomplete training.

Only one chatbot showed strong child protection measures – asking the user’s age, adjusting its responses accordingly, and urging reporting of harmful behavior. This last approach highlights that effective child safety protocols are not only possible, but also essential to any AI companionship application.

Vosloo: Your research offers recommendations to protect children in the context of companion AI apps - can you share a few of these?

Firmino: Governments must develop and enforce age-appropriate design codes, as most companion AI apps lack child-specific safeguards. Safety-by-design should be a standard for AI companion apps, with robust age assurance, removal of guest access, and properly trained AI models for child interaction – regardless of the intended audience, given that children are active online users and easily access platforms.

AI chatbots are not a replacement for professionals or human relationships and should not be marketed as such. App stores also have a critical role to play enforcing their own policies to ensure AI apps meet child safety and ethical standards. Universal AI literacy is essential for children, parents, and educators in an increasingly AI-driven world.

UNICEF is updating its Guidance on AI and Children, including to address the risks posed by AI systems offering companionship. Recommendations include:

• AI chatbots should be developed with robust supervised safety training, transparently and explicitly disclose that they are not humans, and should never be intentionally designed to create emotional dependency.
• Guardrails are needed to identify and stop interactions based on risky content and behaviour, and redirect children who disclose risk of harm to support resources.
• Such AI systems should be subject to child safety laws, and any AI system that manipulates or persuades children resulting in harm must be prohibited – such as stipulated in the European Union’s AI Act.