Short read: Why so many AI researchers think the machines could kill everyone

When we heard from experts that ‘AI will change the world’, no one expected it would mean by them killing all humans! Most industries have their share of critics, but there is something worrying about one whose own builders keep resigning from their jobs because they are frightened of what they are building. But that is exactly what has been happening across the frontier AI labs this year, and Will Knight captures it well in this piece for WIRED. The people walking out are not disgruntled outsiders or reflexive technophobes. They are the researchers who, until recently, were writing the code.

Take Rishub Jain, who left Google DeepMind in June. His worry was not a rogue robot but something quieter. By using AI to speed up the building of the next generation of AI, he felt he was writing himself out of the loop. “AI progress is increasing,” he tells Knight, “and as AI becomes more capable, it poses more risks.” The industry has a name for the process that unnerved him – recursive self-improvement, the point at which AI begins upgrading itself, in a loop, with ever less human supervision.

No lab claims to have built such a self-improving machine yet; it remains theoretical. But the theory is starting to feel uncomfortably close. This month Jacob Coxon quit Anthropic with the warning that AI firms are “racing straight to self-improving superintelligence and gambling with our lives.” A senior Anthropic safety researcher then offered an even blunter estimate: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

A one-in-ten chance of extinction, volunteered by someone whose actual job is safety, is the sort of sentence that would shut most other industries overnight. Nate Soares, co-author of the just-published and bluntly titled “If Anyone Builds It, Everyone Dies,”tells WIRED that the vision of recursive self-improvement “is spooking people,” and that “it’s starting to feel real.” Alignment, the field devoted to keeping a model’s goals in step with ours, was supposed to get easier as the machines got smarter. Instead, Soares says, it is getting harder, and researchers are quietly realising it.

As for how a machine might actually finish off the species that created it, Soares offers a few routes: manipulating people into a catastrophe, an army of killer robots, or an AI wired into a biolab that holds its own off-switch hostage. “We could say we’ll turn it off,” he says, “but it could say, ‘Unfortunately, I have your off switch, which is this super virus.'”

The most damning thread in the piece is not about robots but about incentives. The researchers keep circling the same point: that the companies are “locked in a race to get there first,” as Coxon put it, even as OpenAI and Anthropic barrel towards their respective IPOs. When the reward for moving fast is a market listing and the price of caution is losing the race, a firm’s stated belief in the danger and its actual behaviour can drift a long way apart. It is a governance and incentives problem before it is a technology one.

We would add a note of our own scepticism. This is, by design, a gathering of the alarmed. WIRED itself points to critics who argue the doom talk is “meant to distract us” from AI’s nearer and duller harms – cyberattacks, disinformation, job losses – and even Jain, having quit, has since started a company on the hopeful bet that humans and AI working together can keep models in check. A greater-than-10% extinction estimate is a claim, not a measurement, and some of the people making it have professional and financial reasons to be heard.