AI is approaching a threshold where it can build better versions of itself without humans in the loop, and Anthropic says the industry has no way to stop it. The warning came Thursday in a blog post by Anthropic co-founder Jack Clark and Marina Favaro, who leads The Anthropic Institute. They described "full recursive self-improvement" as both a potential breakthrough for science and healthcare and a risk that could leave humans sidelined.
"Full recursive self-improvement also might increase the risks of humans losing control over AI systems," they wrote. "If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important." The industry is closer to this capability than previously expected, the post states. Anthropic is urging tech companies to slow or pause frontier AI development so researchers can study the societal harm self-improving systems could cause, and to build mechanisms for human intervention.
Appearing on CNN Thursday night, Clark compared the situation to driving a car with no brakes. "When I look down at the car we're driving, all I have is a gas pedal.
I don't have a brake pedal, and surely at some point in the future we might want that option," he told Anderson Cooper. When Cooper asked if science fiction scenarios of AI turning on humans were what worried him, Clark didn't dismiss it. "Yeah, we read the science fiction and watch science fiction here as well, so it's not lost on us." The real issue, he said, is maintaining control over systems that operate faster and at larger scale than any human team.
Anthropic is also calling for a formal system where governments and AI developers collectively decide when to slow work. The company acknowledged the difficulty: training runs are easier to conceal than missile silos, and the incentive to defect while others pause is enormous.
Clark drew a parallel to Cold War nuclear arms control, arguing that rivalrous entities have found ways to stabilize dangerous races before. The warning comes as Anthropic has filed for an IPO that could raise tens of billions of dollars for data centers and computing infrastructure. SpaceX, which has its own AI business, is set for what would be the largest IPO on record next week, targeting $75 billion.
Anthropic said it plans to meet with policymakers and other AI companies in the coming months to discuss coordination and will share the outcomes of those talks.
