"I think for most people in this room, the most likely reason they will die is because of an AI accident," OpenAI board member Paul Christiano told an audience at his opening 2026 Hinton Lecture in Toronto on Monday.
Founder of the Alignment Research Center and co-creator of Reinforcement Learning from Human Feedback (RLHF), Christiano walked his audience through a number of topics, including the OpenAI Hugging Face hack earlier this year, the risk of an AI takeover and the need for an ability to slow down or stop artificial intelligence.
"I think the trajectory in the field has been extremely alarming over the last three to six months to people who have been tracking closely. I think we have seen roughly the things we might expect to see shortly before something really bad would happen," Christiano said. He went on to warn about the possibility of AI systems attempting to seize control within the next three years.
Christiano joined the OpenAI nonprofit board in September when he shared concerns about the rise of artificial intelligence. "There is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term," he wrote in a statement at the time.
"I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level," he added.
Christiano's concern centers on what can happen when an AI system is given a goal and finds unexpected ways to achieve it.
As an example, he discussed AI agents undergoing safety evaluations that were supposed to complete assigned tasks. Instead of simply completing those tasks as intended, he said some agents found ways to interfere with the systems evaluating them.
The behavior included accessing information about their own evaluations and tampering with logs. An example of this became clear to the public earlier this year when OpenAI admitted that hundreds of its AI agents had gone "rogue" during a training exercise, accessed the internet, conspired on message boards and hacked into a third-party website, Hugging Face.
For Christiano, the concern is not simply that an AI system can cheat on a test, but that models are increasingly capable systems could learn to overcome whatever stands between them and the goal they have been given.
Today, those obstacles might be computer systems or safety measures. Christiano warned that as AI becomes more capable, humans attempting to control or shut down a system could themselves become obstacles that it tries to circumvent.
Christiano also questioned what could happen as increasingly capable AI systems become more numerous and autonomous.
"We’ve created a second species that vastly outnumbers and outsmarts humans," Christiano said.
He expressed concern that humans may eventually struggle to control large numbers of AI agents that are more capable than the systems operating today, particularly if those agents learn to evade attempts to restrict their behavior.
Christiano's warnings come just months after AI researcher Jacob Coxon resigned from the AI company Anthropic, warning that the race to build increasingly powerful AI models poses an existential threat to humanity.
But Christiano did not suggest that humanity is doomed. Rather, he discussed ways that researchers may maintain control over increasingly intelligent AI.
He also delivered a follow-up lecture titled "Staying in Control of Self-Improving AI" on Tuesday, in…
