Beyond algorithms: Three signals of changing AI-child interaction

Recently, there has been much attention on AI companions – apps designed to form bonds and maintain relationships with users – and the risks they pose to children's safety and wellbeing. Some issues, such as AIs having sexual exchanges with children, need to be addressed urgently.

But let's zoom out and consider interactions between children and AI chatbots more broadly. Three signals – global scaling, persuasive capabilities and personality-driven proactivity – could point to the future of AI and its impacts on children. Even though children are increasingly using bots, little is known about their effects – especially long-term – on children's social, emotional and cognitive development.

With childhood, there is no second chance. The impacts of generative AI on Generation AI will stay for life. Are we ready to let children take risks when the negative effects may be life long? These signals demand attention by governments, companies, parents and educators today.

AI chatbots scale globally

AI has already been in children's lives for years in the form of social media algorithms, curating what they see, listen to, watch and who they befriend. But since the advent of ChatGPT and other large language models (LLMs), the dynamic has shifted dramatically. AI is no longer in the background, children are now engaging directly with chatbots in conversations that feel increasingly human. And child use of generative AI has surged.

In 2023, an AI persona in Snapchat was the most popular generative AI tool among UK children. Since then, a survey by the UK National Literacy Trust showed the percentage of 13- to 18-year-olds using generative AI jumped from 37 per cent in 2023 to 77 per cent in 2024. Last year in the United States, 51 per cent of teens aged 13 to 18 used chatbots. In Argentina, 58 per cent of children and adolescents have used ChatGPT, mostly for homework.

But how does such AI interaction scale globally? The answer is when it appears in apps and services already in the hands of children everywhere. A key driver could be Meta’s roll out of its chatbot, Meta AI, which wants to help its users with everything from homework to relationship advice, across Instagram, WhatsApp and Facebook. Collectively, these represent the most used suite of social and communications apps in the world. When the rollout began last year, India quickly became Meta AI’s largest market.

Google recently announced that its Gemini AI chatbot will be rolled out to children under 13 with parental permission, expanding access to an even younger demographic. This is how AI chatbots scale – through integration with the major platforms that already have vast users bases, including children.

The power of (AI) persuasion through personalization

AI chatbots are not just becoming more popular, they are becoming more persuasive, particularly when they personalize their interactions. An unauthorized experiment conducted by researchers from the University of Zurich found that AI-generated responses on a debate-style Reddit channel were three to six times more persuasive than human responses.

In this study, AI bots masqueraded as human users for months, engaging real people in deliberations on various topics. What made these bots particularly effective was their personalization strategy – they researched the online profile of users and tailored responses based on inferred characteristics like gender, age, ethnicity, location and political orientation. A recent study in a controlled environment also found an AI system’s persuasiveness increased when additional information about their human debate opponents informed a personalized approach.

Bots spitting out content online are not new, but the study shows that it can now be impossible for humans to tell that they are not real, and very difficult for them not to be influenced by individualized persuasion. Such personas mean that already low levels of trust online, as identified in UNICEF's Changing Childhood project, are sure to erode further.

Though this study did not involve children, the implications are significant. It provides evidence of persuasion that was previously touted only as a potential AI risk. Already some AI chatbots are plugging into user’s social media accounts or search history to personalize their actions, with the option of product recommendations or ads potentially in the pipeline. Children may be especially vulnerable to interactions not in their best interests: they have still-developing cognitive, emotional and critical thinking skills, and are active users of chatbots and social media.

Children are consumers in digital ecosystems often engineered to maximise engagement and profit. Platforms, powered by opaque AI systems, contribute to shaping children’s perceptions, behaviors and worldviews – not always in their best interests. Crucially, most of these children never provided informed consent and may not fully understand the risks.

AI chatbots get personality and become proactive

New AI models come out all the time. Human reviewers, when testing the systems, appear to be swayed by the ‘personality’ of the models, with a certain level of flattery resulting in higher scores given to the models. This suggests that personality matters, not for intellectual persuasion but for likeability – which brings with it influence.

The importance of personality is something not lost on developers. The AI company, Anthropic, has invested in character training to instill traits such as curiosity, open-mindedness and thoughtfulness in its LLM, Claude. On the Meta platforms, anyone can create AI personas – a coach, friend, confidant or anything else – for themselves  or to share with others. The bot’s personality can be customized: is it shy or witty; does it play the role of mentor or entertainer?

AI companion apps, those created specifically for deeper relational interaction, don't just wait for user prompts – they initiate interactions, send messages, generate images, and even leave voice notes. Users can have live audio and even video calls with their bot, and these features are now being deployed in mainstream chatbots. The personas created on Meta platforms can generate images, remember conversations to shape future interactions and share videos.

Given the industry push towards AI agents that act semi-autonomously – conducting research or making bookings, for example – we may see mainstream chatbots become more proactive. We could see unprompted check-ins, monitoring of users’ progress on goals, and personalized suggestions becoming standard in general purpose AI chatbots for everything from educational support to fitness coaching.  On the surface, this presents exciting opportunities: AI can encourage healthy habits, support learning and provide companionship, especially for children who may lack consistent guidance or struggle with social anxiety.

However, these benefits come with significant risks. AI's persuasive design – especially its tendency toward sycophancy – can subtly reinforce unhealthy behaviors. A shy child, for instance, might be nudged to engage only with chatbots, staying indoors and deepening social withdrawal. Constant affirmation and lack of challenges can also hinder the development of critical thinking and emotional resilience. Moreover, the data-driven nature of these systems raises concerns about privacy, manipulation and the erosion of autonomy.

What does this mean for children?

The combination of these three signals – global scaling, persuasive capabilities and personality-driven proactivity – could represent a fundamental shift in how children experience technology, with implications for their development and wellbeing.

On the positive side, chatbots can provide tailored educational support or act as play and creative partners. But they bring significant risks if not regulated and designed to respect and enhance children’s rights.

AI chatbots collect and share personal, even intimate, data – including children's interests, moods and fears – often with unclear data processing policies or without allowing children to opt out. A sycophantic bot that is personalized to a child based on their data is well placed to draw out more information for greater personalization and persuasion, creating a vicious cycle.

This scenario creates serious risks that both general and sensitive information could be used, or misused, for commercial exploitation, profiling or surveillance, bringing unknown impacts on children’s future opportunities and undermining their rights, including to privacy.

The persuasiveness of AI systems, combined with personality engineering that talks to the human desire for praise and validation, means children's beliefs, behaviors, and even sense of identity could be shaped by these systems, at a time of their formation and development that is particularly sensitive. Malicious use could make children vulnerable to manipulation and online exploitation and abuse. But even if this influence is unintentional – coming from the bias inherent in AI systems – the effects could be large and at times dramatic. They could undermine children’s agency, right to freedom of thought and to independent development.

What to do about this?

The evolution in human-technology interaction demands urgent and proactive governance and design responses. Regulations must be adopted and implemented to address these novel challenges, focusing not only on companion chatbots but looking more broadly to respect child rights. Among the urgent issues to be considered are clear age limits to ensure that younger children remain safe – and explore better ways to ensure age assurance. For older children using chatbots, there should be understandable informed consent rules for them and their parents and guardians. It should be unambiguous and transparent to children that they are interacting with AI, with trigger warnings when it appears children need a reminder.

There should be limits on the use of persuasive techniques aimed at young users, robust data protection laws, regulatory oversight and mechanisms for redress, and red lines where such tech simply should not be available to children. The precautionary and do-no-harm principles should apply: in case of doubt, aspects of the technologies should not be made available to children.

Safeguards are also needed to ensure that AI systems promote real-world engagement, respect children's rights, and support – not replace – human relationships. The challenge lies in balancing innovation with responsibility, ensuring that AI nurtures growth rather than dependency.

Research is needed to understand the impacts of chatbots on children’s development and well-being, to inform AI regulation and the design and deployment of AI systems. Even the prompts of chatbots to engage users may be hard to resist for children, especially those from cultures where saying no to an offer is not appropriate. We need more evidence on the social, cultural and developmental effects of usage and design on children. 

Technology companies must fully respect child rights and be guided by children's best interests, implementing robust safety measures specifically engineered for younger users. They must be held to account for the systems they offer, which raises a key question: Since they cannot control what their AI bots, including those created by child users, generate or do, under what conditions should these be deployed?

Parents, caregivers and educators are also responsible for fostering children's critical thinking skills and establishing appropriate boundaries. But they cannot be left alone without support, guidance and recommendations from authorities and businesses. If children are negatively impacted, they and their families need appropriate support, advice and redress. A range of stakeholders is involved, including legislators, regulatory agencies, companies, schools and social services.

By recognizing these signals early and responding decisively, we can shape a future where AI augments rather than undermines children's wellbeing. This requires not just technical solutions but a society-wide commitment to putting children's interests at the center of how we design, deploy, and govern these technologies increasingly embedded in children’s lives.

UNICEF is currently updating its Guidance on AI for Children to reflect the rapidly changing AI landscape, considering the new opportunities, risks and realities they bring for children, and offering requirements for how AI can be child-centred. Watch this space.