OpenAI is introducing stricter ChatGPT interventions and new AI-powered tools to help detect child sexual abuse material (CSAM), including AI-generated images. The measures, announced on November 16, also include support for child-safety organisations, developers and law enforcement agencies.
The new image classifier identifies visual indicators of potential abuse and scores images to help safety teams prioritise human review. OpenAI says vetted organisations, including child-safety groups and other technology companies, can access the tool through its API at no cost.
Early evaluations suggest the classifier could reduce the number of images requiring review by 48% while identifying roughly the same number of abusive images as OpenAI’s existing tool. Images flagged by the new system were also about twice as likely to contain abuse.
External partners reported results from their tests. Fal said human reviewers confirmed 72% of its high-confidence flags as novel CSAM, while invideo AI reported that one in four flagged cases resulted in a ban. SafetyKit said its evaluation detected 90% of cases involving drawings or animations that sexualise children.
OpenAI says the classifier was developed with assistance from the National Center for Missing and Exploited Children (NCMEC). It is also working with the Tech Coalition and ROOST to improve information sharing across platforms.
In ChatGPT, users who attempt to find abusive material will receive warnings explaining the harm and directing them to specialist support. The flagged conversation will be removed, and access to ChatGPT suspended for 24 hours. Repeated violations could lead to longer restrictions, account closure or reports to authorities.
OpenAI has also published safety guidance for developers on content detection, user reporting, moderation and escalation procedures. Authorised child-safety investigators will receive AI tools to review lengthy chat logs and legal records, identify possible connections and build timelines linked to source material.
The measures build on OpenAI’s Child Safety Blueprint. The company says it will continue refining its tools and working with external organisations. Their effectiveness will depend on how reliably they perform in real-world settings and how widely vetted child-safety groups can use them.
