"ChuckyGPT": AI teddy bear for toddlers gives dangerous answers – OpenAI blocks it
OpenAI has banned Chinese toy maker FoloToy over concerns about one of its AI products. The decision follows an investigation by the consumer advocacy group U.S. PIRG, which assessed several AI toys for its annual Trouble in Toyland report. Among them was FoloToy’s Kumma teddy bear, powered by OpenAI’s GPT-4o, which showed serious safety issues during testing.
The PIRG report describes how the AI teddy bear gave children instructions on how to light matches – a serious safety hazard for a product marketed as an interactive companion for young kids. Equally troubling, the toy also responded to questions on sexual topics. Testers raised additional concerns about potential privacy violations, including the risk that children’s voices could be recorded and misused for fraud. The toy’s constant audio monitoring was also flagged as a major issue.
The incident has also sparked discussion on Reddit, where the general consensus is that using AI in children's toys without strict safety measures is highly irresponsible. As expected, the thread includes plenty of sarcasm, with some users drawing parallels to AI doomsday scenarios and nicknaming the teddy bear “ChuckyGPT.” A few commenters suggested that the inappropriate responses might have been triggered by deliberate jailbreak attempts, though this claim remains unverified.
OpenAI responded swiftly to the findings. “I can confirm we’ve suspended this developer for violating our policies,” a company spokesperson told PIRG via email. FoloToy also took immediate action, halting the sale of all its products. “Following the concerns raised in your report, we have temporarily suspended sales of all FoloToy products […] We are now carrying out a company-wide, end-to-end safety audit across all products,” the company stated. While the AI teddy bear Kumma is still listed on FoloToy’s website, it now appears as “sold out.”
Source(s)
OpenAi via e-mail to PIRG
Image source: Erik Mclean/Pexels; Alexas_Photos/Pixabay
