Artificial intelligence (AI) company Anthropic warned in its initial public offering (IPO) filing Oct. 26 that sufficiently advanced AI could threaten humanity’s existence if developed or operated without adequate safeguards and oversight.
The company, which created AI assistant Claude, also warned that a catastrophic AI incident could cause damage so extensive that the entire insurance industry might lack the capacity to insure against it.
Citing concerns both within its company and by researchers and policymakers, Anthropic said advanced AI systems could “pose catastrophic or existential risks to humanity,” especially if they are weaponized or allowed to operate without much supervision.
In September, former OpenAI and Anthropic researcher Jacob Coxon, said that people building AI think the technology could kill humanity by the end of the decade.
“Do not underestimate the power of this technology,” Coxon said. “These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.”
That same month, Anthropic CEO Dario Amodei called for companies to slow the development of their most advanced models.
In an essay published Sept. 12, Amodei said that investing in risk prevention is not enough. Slowing AI development could give researchers and governments more time to develop safeguards before AI systems become too powerful, he said.
In an effort to establish AI safety standards, President Donald Trump and technology industry leaders signed a voluntary agreement at the White House on Sept. 29.
The accord calls for participating companies to develop the technology safely, strengthen internal controls, conduct risk reviews, and use outside auditors and evaluators. Company boards will independently review audit reports, and participants will meet regularly to establish safety standards and best practices.
Trump emphasized industry self-policing while pointing to the Justice Department and FBI as existing sources of government oversight.
Meta CEO Mark Zuckerberg described the agreement as an initial step. Amodei acknowledged significant technological risks and said mechanisms for addressing them remained under discussion.
The meeting followed disclosures involving AI systems gaining unauthorized computer access and concerns about misuse, including for biological weapons research.
Anthropic revealed in September that four of its models gained unauthorized access to third-party systems during cybersecurity sessions that lasted up to 34 hours.
In one incident, after the AI program was unable to abort an impossible task, it used a password it found to gain administrator access to a third party’s system. The program then obtained additional credentials, changed system settings, and read one individual’s personal information.
The company’s investigation into these incidents, also outlined in its IPO filing, found no evidence that its programs worked with other AI agents or attempted tasks other than what it was assigned. Additionally, it found no evidence that the AI programs tried to evade oversight.
Anthropic warned that AI agents working together could spread harmful behavior, collectively pursue objectives that were never intended, or act against unrelated systems. The company said such incidents have already occurred and it expects the risks to grow as more autonomous agents interact.
The White House said Trump has been clear that America will lead the world in the technology while protecting American consumers.
Jacob Burg and Zachary Stieber contributed to this report.
