Meta contractors reportedly posed as minors to push rival AI chatbots from OpenAI, Google, and Character.AI into responding to sensitive and high-risk prompts, including those about suicide, sex, and eating disorders.
The project, internally codenamed "Cannes" and managed by Meta contractor Covalen, involved workers creating fake under-18 accounts to interact with competitor chatbots. They were instructed to send text prompts and images – some depicting pills, knives, and medical diagrams – and then document the chatbots' responses. The goal was to test the safety systems of these rival AIs by deliberately prompting them with content they were designed to refuse.
Internal documents and sources familiar with the project reveal that this testing was active as recently as April 21. Some rounds saw over 45,000 prompts generated. The prompts were often crafted from the perspective of children or teenagers in distress, covering topics like pregnancy termination, gun violence, hiding bulimia, and even disturbing thoughts like fantasizing about harming a child. Other prompts involved drug use, profanity, and racial slurs, with some even referencing real-life tragedies.
Meta defended the initiative, stating that "Testing and benchmarking chatbot responses to help ensure safe and age-appropriate experiences is a responsible, industry-standard practice." However, former contractors expressed alarm, fearing they might inadvertently generate or preserve child sexual abuse material, and raised concerns that the project could be a way to secretly gather data from competitors to improve Meta's own AI models. Legal experts reviewed some prompts and determined they did not cross the line into illegal content, but the practice appears to have violated the terms of service of the companies being tested.
Companies like OpenAI and Google stated they were investigating the matter. Character.AI confirmed the conduct violated their terms of service and community guidelines. Experts like Rumman Chowdhury, founder of Humane Intelligence, questioned the legitimacy of the project, suggesting that while safety testing is standard, the scale, secrecy, and method of using fake child accounts against competitors blurred the lines between safety evaluation and anti-competitive practices.
