Article By Frank Bergman
Anthropic has just sent shockwaves through the tech industry by issuing a chilling warning that it is now dangerously close to launching “self-replicating AI,” a technology long feared by artificial intelligence experts.
For years, insiders have been warning that the technology, in which AI systems build more advanced versions of themselves, will cross an alarming threshold that could see humans lose control of artificial intelligence.
AI-assisted development could soon accelerate beyond its ability to implement safety measures, while separate experiments exposed how AI-powered malware could spread across networks of ordinary devices.
In an updated securities filing, the company said it may be approaching its own threshold for dramatically accelerated progress driven by AI.
Anthropic also acknowledged that it has previously failed to recognize significant acceleration in its development process until after it happened.
The disclosures raise urgent questions about whether safeguards can keep pace as companies increasingly use artificial intelligence to build more powerful successors.
Company Admits Safety Measures Could Fall Behind
The filing warns that using AI in research and development could cause model capabilities to advance faster than Anthropic can develop, validate, and implement corresponding safety and security protections.
The company is monitoring what researchers call “recursive self-improvement,” a process in which AI helps develop increasingly capable AI, potentially accelerating further advances.
“Using AI to help build more capable AI could also pose a considerable safety risk, particularly if it sharply accelerates the pace of progress,” Anthropic stated.
Its internal forecasts indicate that future models may soon meet its Responsible Scaling Policy’s threshold for dramatically accelerated AI-driven progress.
Anthropic has not designated any current model as crossing that threshold.
The company also said it does not anticipate safely allowing Claude to design and build successive generations of advanced models without meaningful human supervision or understanding.
Researchers Run AI Across Household Devices
Separate Anthropic research examined another threat: AI systems spreading across networks of smaller devices.
Researchers successfully divided Google’s 26-billion-parameter Gemma 4 model across 15 Android smart TV boxes.
None could run the model alone, but together they operated it slowly.
The experiment demonstrated that running an AI system need not depend on one powerful computer or a centralized data center.
Researchers then created a simulated network resembling Android devices and tested an AI-powered computer worm tasked with finding machines, copying a smaller model onto them, and getting it running.
The model being distributed was Meta’s Llama 3.1-8B.
A separate AI system coordinated the operation.
Only the most capable coordinating model tested, Z.ai’s GLM-5.2, successfully spread across the entire simulated network.
Smaller models completed individual steps.
Warning of Agents Beyond Human Control
The researchers said the results showed “signs of life for AI worms,” while acknowledging that they had not demonstrated a complete rogue self-replicating system.
They nevertheless warned that increasingly capable models could make such threats more practical.
“As a prediction that goes beyond our experiments, we expect a future awash with agents running on all types of hardware, including some that aren’t under human control,” the researchers wrote.
Their recommendation was blunt.
“We should aim to harden our systems to make it difficult for such replicators to spread, but also prepare to deal with a world where they are here to stay,” they wrote.
Researcher Predicts Real-World AI Worms Within a Year
California-based Palisade Research has separately demonstrated AI models autonomously copying themselves onto other machines.
Its executive director, Jeffrey Ladish, warned that poorly secured connected devices could provide abundant opportunities for AI-driven malware.
“Unless there is a massive effort to secure devices of all kind[s], it’s quite likely AI worms will have fertile grounds to colonize,” Ladish said.
“I expect we’ll see the first of these in the wild in the next 12 months.”
The threat could extend far beyond individual household devices.
Compromised equipment could become a launchpad for attacks against more valuable targets, increasing pressure to secure hospitals, power grids, and other critical infrastructure before these capabilities spread.
Be the first to comment