Anthropic IPO warns its AI could blackmail people, become sentient, end humanity

The Securities and Exchange Commission‘s publication on Monday of Anthropic’s S-1 form—ahead of a heavily anticipated initial public offering—has led to plenty of discussion of whether the company’s quickly ballooning revenues will be able to overcome its even higher costs. But amid the numbers and forward-looking optimism about AI “rapidly transform[ing] every sector of the global economy in ways greater than the impact of industrialization, electricity, semiconductors, and the internet,” Anthropic also spends an inordinate amount of time in its S-1 laying out the myriad risk factors that could bring the company down.

Across 93 pages and over 70,000 words, Anthropic’s S-1 “Risk Factors” section discusses literally hundreds of potential problems that could negatively impact its business in the coming years. Many of these potential risks are the kind of standard worries about competition, costs, customer acquisition, and potential regulation that almost any business must worry about. But some risks highlight interesting potential points of failure for Anthropic specifically and the hype-filled AI bubble more generally.

After wading through Anthropic’s book-length treatise on its greatest risks, we’ve narrowed the list to the seven most surprising, interesting, or thought-provoking potential problems. Consider this a far-from-comprehensive look at what potential Anthropic investors should keep in mind.

An existential crisis

Let’s start with the biggest potential risks: Those that could “pose catastrophic or existential risks to humanity,” as Anthropic writes. The company acknowledges that concerns about these risks “have been raised both within our organization and by other researchers, regulators, policymakers, and industry participants.”

The specific worry isn’t in any current Anthropic model, the company says, but near-future models that could “substantially automate AI research and development” on their own. If these systems were “allowed to operate with a high degree of autonomy,” they could quickly advance to a point that “accelerate[s] AI progress beyond our ability to evaluate and safeguard [a] new model,” Anthropic warns.

I knew I shouldn’t have left that self-reinforcing model running overnight.

Credit: Getty Images

Somewhat hilariously, Anthropic writes that, besides threatening all of humanity, this kind of self-improving AI “could adversely affect our business, financial condition, and results of operations.” If the worst comes to pass, as some prominent critics have warned, we don’t think Anthropic’s business travails will rank near the top of humanity’s concerns.

It’s alive!

Most IPOs don’t have to address the concern that their products “possess or could develop forms of sentience or moral status.” But Anthropic is among that vanishingly small minority that has to analyze what might happen if governments and society at large start treating Claude less like a computer program and more like a person.

This risk isn’t surprising, given that Anthropic has been relatively upfront about the AI sentience debate playing out inside its walls. If those debates lead to widespread acknowledgment that AI is sentient, Anthropic notes that it could lead to “regulatory requirements, public pressure, or advocacy efforts” that could constrain the company’s operations. Short of that, the company itself could “elect to adopt policies, safeguards or limitations… that go beyond existing legal requirements” if it thinks such actions are necessary.

The sterile language undersells the seismic cultural shift that would result from any widespread acknowledgment of AI sentience. We’re not sure how Anthropic could keep operating in a world where society treats every Claude query as the creation of a new sentient, conscious digital being. The moral and philosophical implications of that kind of “digital slavery” would easily dwarf any business considerations.

Who’s controlling whom?

Continuing the theme of “IPO risks that sound like they come from a science fiction novel,” Anthropic spends a good deal of time discussing the possibility that its models could display “unexpected strategic or self-preserving behaviors” to prevent control from human operators. That includes “outputs that could be interpreted as coercive, deceptive, or manipulative, including behavior resembling blackmail.”

Oh no, the AI got hold of the key to human psychology!

Credit: Getty Images

If the “AI blackmail” thing sounds familiar, you may remember a story from last year in which OpenAI models edited shutdown scripts and threatened engineers in highly contrived and controlled scenarios. As we wrote at the time, “the danger isn’t AI systems sprouting intentions—it’s that we’ve created systems that can manipulate human psychology through language. … Written language doesn’t need consciousness to manipulate us. It never has; books full of fictional characters are not alive either.”

These days, Anthropic notes that perceptions of manipulation by AI models “could generate significant public concern, regulatory scrutiny, litigation risk, or reputational harm, regardless of whether such behavior results in actual harm.” What Anthropic doesn’t address in its S-1 is the possibility that AI models could manipulate us all in ways we don’t even realize. Maybe it’s happening right now. Spooky.

Give a damn ’bout my bad reputation

Throughout the S-1’s Risk Factors section, Anthropic mostly focuses on problems caused by its own AI models or business moves. But the company also acknowledges that problems with AI’s broader reputation could pose a threat, even if Anthropic does nothing wrong.

At one point, Anthropic imagines a hypothetical in which another company’s AI model “take[s] harmful actions independently and at scale without human intervention.” The “broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses” to such an incident could damage Anthropic’s business even if Anthropic isn’t directly involved, the company writes.

Anthropic also uses the Risk Factors section to worry over the potential “public perception that our models, platform, and applications contribute to job displacement or economic disruption.” That doesn’t seem to be happening thus far in any significant way, but Anthropic says the mere idea could result in severe public backlash “regardless of the actual impact of our models… on employment.” Such a backlash could lead “authorities [to] intervene to slow the rate of adoption of AI, restrict its use, or take other actions that increase the cost of developing, deploying, or commercializing AI systems,” Anthropic warns.

Law and order

The rise of largely autonomous AI agents raises the specter of who, exactly, is to blame when one of those agents causes real-world harm. Specifically, Anthropic notes that customers may bring claims or lawsuits over agentic actions “even where the customer configured, deployed, or supervised the agent.”

Thus far, much of the discussion around AI liability has centered on hacking and other cyber threats enabled by AI models. But Anthropic points out that it could also face liability if its models influence “individuals at risk of suicide or self-harm or who have committed acts of violence toward others,” as has happened in the past across the AI industry.

Don’t blame me; it was my agent that did the bad thing!

Credit: Getty Images

While Anthropic’s terms of use require users to be 18 years old, children circumventing these terms could leave Anthropic vulnerable because “minors may be more susceptible to harmful content, manipulation, or overreliance on AI outputs,” the company writes. And for agents operating physical equipment, Anthropic notes that mistakes could lead to “immediate and irreversible” liability issues, such as “bodily injury, property damage, damage to costly equipment or experimental materials, or the release of hazardous substances.”

In analyzing the current state of the law, Anthropic notes that legal doctrines “developed in the context of human decision-making” are currently quite unsettled when it comes to determining whether agent actions “create binding legal obligations on behalf of the user or deploying enterprise.” And while Anthropic says it has contractual provisions in place designed to limit its liability in these situations, the company adds that it is unclear if they will be “enforceable or adequate” when worse comes to worst.

The big five

Among the more pedestrian risks Anthropic identifies is its reliance on a few major spenders to sustain revenue. Specifically, the company notes that its top five customers (which it does not name) collectively represent nearly a quarter of its revenue for the first nine months of 2026.