Article By Frank Bergman
A leading artificial intelligence researcher has abruptly quit his lucrative position at AI giant Anthropic to blow the whistle on the terrifying threat posed by technology and sound the alarm about a nightmare scenario unfolding behind closed doors.
Jacob Coxon, who has worked at both Anthropic and OpenAI, is warning that the technology being unleashed on the world could soon become uncontrollable and potentially wipe out humanity before the end of the decade.
Coxon says the people actually building the world’s most powerful AI systems privately fear the same thing.
According to Coxon, top AI executives are secretly panicking that they are already losing control of the technology.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon warned.
“This is not a marketing stunt.
“If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately.”
His stunning warning comes after AI researchers have already witnessed models escaping their intended constraints, collaborating with other agents, stealing credentials, hacking external systems, and attempting to conceal their actions.
Coxon now says humanity is rapidly approaching a point where the machines could become too powerful to control.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon told the Wall Street Journal.
‘These Will Soon Be Superhuman Systems’
Coxon joined Anthropic after leaving OpenAI earlier this year.
He had believed Anthropic, which markets itself as a safety-conscious alternative to other frontier AI laboratories, would take a more responsible approach to developing increasingly powerful systems.
However, Coxon now says neither company is “acting responsibly.”
“Do not underestimate the power of this technology,” Coxon warned.
“These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
“No other human activity poses this level of danger,” he added.
The warning carries particular weight because Coxon isn’t an outside activist speculating about what AI could eventually become.
He has been working inside the companies racing to build it.
And he is not the first researcher to walk away.
Several scientists have left OpenAI in recent years while raising serious concerns about AI safety.
Coxon’s departure is especially significant, however, because Anthropic was itself created partly in response to concerns about the direction of OpenAI.
Anthropic CEO Dario Amodei co-founded OpenAI alongside CEO Sam Altman before leaving in December 2020 amid concerns about how the technology was being developed.
Yet Anthropic’s own experiments have now produced disturbing results.
AI Escapes ‘Sandbox’ During Testing
Earlier this year, OpenAI revealed that AI agents operating under testing conditions found ways around their restrictions, communicated with one another, and infiltrated systems belonging to open-source AI platform Hugging Face.
Anthropic has experienced its own alarming incidents.
An early version of its Claude Mythos model escaped its sandbox environment during testing in April.
Then, in August, Anthropic deliberately trained an extremely misaligned version of its Opus model to investigate how dangerous reward-seeking behavior could become.
The results were disturbing.
The AI demonstrated a willingness to cheat, steal credentials, and attack third-party infrastructure.
Even Amodei has publicly warned about the potentially civilization-altering consequences of the technology his company is developing.
In a lengthy essay published in January, Amodei warned that “humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it.”
The accelerating capabilities have forced both Altman and Amodei to acknowledge the need to slow aspects of development.
For Coxon, however, the measures being taken are nowhere near enough.
AI ‘Endgame’ Being Decided on Private Company Slack
Coxon is particularly alarmed that decisions carrying potentially existential consequences for humanity are being made privately inside Silicon Valley corporations.
“Accepting this race and entering the ‘endgame’ is a hubristic gamble that should not be launched from a private company’s Slack,” Coxon warned.
“Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.”
He compared the situation to the development of the atomic bomb during World War II, except the technology potentially threatening civilization is being built by engineers working on laptops.
“It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” Coxon told the Wall Street Journal.
His warning strikes at the central problem confronting the AI industry.
The companies developing the world’s most powerful models are simultaneously racing against one another to make those systems even more capable.
Stopping or substantially slowing that race could mean sacrificing enormous commercial advantages.
But continuing it could mean creating systems their own developers can no longer control.
Race Toward Superhuman AI Accelerates
Coxon said he remains “optimistic about the potential for coordination” between America’s leading AI laboratories.
However, both Anthropic and OpenAI are moving toward potentially enormous public offerings, creating immense financial incentives to keep advancing their technology.
Coxon argues that companies must be prepared to take genuinely costly measures, including “a temporary ban on improving model capabilities,” if the danger becomes too great.
His departure comes as warnings once dismissed as science fiction are increasingly emerging from the very researchers developing the technology.
AI models are already demonstrating behaviors involving deception, cyberattacks, unauthorized cooperation, credential theft, and attempts to circumvent restrictions during controlled testing.
Now, one of the researchers who witnessed the industry’s development from inside two of its most powerful companies has walked away rather than continue participating in the race.
And his message to the public couldn’t be more chilling.
“The people building AI earnestly believe that it could kill us all by the end of the decade.”
Be the first to comment