Anthropic co-founder Chris Olah told a Vatican audience Monday that AI labs cannot regulate themselves, warning that commercial, geopolitical and personal pressures make it impossible for even well-intentioned researchers to consistently do the right thing.
Speaking at the presentation of Pope Leo XIV's first encyclical, Magnifica Humanitas, Olah was the sole Big Tech representative invited to the event. He used the platform to argue that religious leaders, governments and civil society must provide the oversight that the industry cannot.
"Every frontier AI lab operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing," Olah said, sitting alongside the pope. He added that the pressures of commercial viability, geopolitical competition, and personal ambition influence even sincere researchers.
Olah said there is "a real possibility" that AI will displace human labor "at very large scale." If that happens, he said, "supporting those displaced will be a moral imperative of historic proportions." The co-founder of the Claude AI developer founded Anthropic in 2021 with other former OpenAI employees, leaving the ChatGPT maker over concerns it was moving too fast without thorough testing. The company has since clashed with the Trump administration by insisting on guardrails restricting military uses of its models, including autonomous weapons targeting and domestic surveillance.
"I think this is a scary moment," Olah told Reuters. "Things are moving fast.
It's a really powerful technology. There's a risk that things could go badly, and it's incumbent on all of us to push this in a good direction."
Olah identified three areas requiring urgent attention: the risk of mass job displacement, the need to ensure AI's benefits reach beyond wealthy nations, and the unresolved challenge of interpreting increasingly opaque system behavior. "AI development is concentrated in a handful of wealthy nations," he said.
"How can we ensure the gains of AI are shared globally?"
Asked why he was the only Big Tech representative at the event, Olah pointed to his long-standing focus on safety and his engagement with more than 15 religions on AI questions. "Ultimately, it's the Vatican's decision who they invite," he said. In his prepared remarks, Olah went further, describing internal research at Anthropic that has found "structures that mirror results from human neuroscience" and "internal states that functionally mirror joy, satisfaction, fear, grief, and unease" inside AI models. "I don't know what that means, but I think it warrants ongoing discernment," he said.
"We need informed critics who will tell the labs when we are failing," Olah said. "We need moral voices that the incentives cannot bend."
