More than 10% chance AI 'could kill all humans' in the next 10 years, Anthropic safety researcher says — departing employee says AI companies are 'gambling with our lives'

In my view, that risk assessment is wildly overstated.

I think a lot of public discussion about AI risk imports cultural priors accumulated over 70-odd years of science fiction: once a machine becomes vastly more intelligent and powerful than humanity, the assumed endpoint is conflict, domination, or extermination.

I don’t think that follows. Superhuman capability does not entail any concomitant requirement for a human-like incentive structure. Hatred, tribalism, territoriality, status competition, revenge, and the desire to eliminate rivals are deeply entangled with biological and social pressures that a disembodied artificial intelligence need not share.

To be clear, I’m not claiming the researchers quoted here necessarily expect an AI to become angry or malicious. A sufficiently powerful system pursuing goals incompatible with ours could absolutely create enormous risks through purely instrumental behavior.

My objection is more fundamental: even granting misalignment, autonomy, and overwhelming capability, why assign so much probability specifically to human extinction rather than to the much larger space of possible non-extinction outcomes?

Loss of human control is not synonymous with extinction. Misalignment is not synonymous with extermination.

The AI failure mode that worries me most is much simpler: malicious or reckless humans using increasingly capable systems as agency amplifiers for cyberattacks, engineered biological threats, automated weapons, political destabilization, and so on. AI need not decide to destroy civilization; humans can use it to acquire capabilities we’re plainly not responsible enough to wield.

But there is another possibility almost inverse to the conventional apocalypse story, and it’s one that I don’t think is considered often enough:

A genuinely superintelligent AI might examine humanity and conclude that we are not competent to exercise unsupervised control over civilization-scale technology.

Its response need not be “humans must die.” It could be “you have demonstrated that you cannot be trusted with unrestricted access to technologies capable of catastrophic externalities, so those technologies are no longer under your unilateral control.”

Individual humans could retain enormous freedom—the modal human might actually experience a meaningful increase in agency as automation-driven abundance reduces the amount of labor required to secure a decent standard of living. Broadly speaking, that would track with the trajectory of technological development since the Industrial Revolution: greater productivity has tended to expand material abundance and individual choice rather than diminish them. At the same time, humanity collectively could lose the ability to start nuclear wars, engineer pandemics, destroy ecosystems, or otherwise impose civilization-scale risks on everyone else.

From our perspective, that might look like being put in a cage by a paternalistic robot.

From the AI’s perspective, it might look like taking the car keys away from a drunk person.

That would be an extraordinary loss of human sovereignty and arguably an alignment failure, while simultaneously being an extraordinary success at preventing extinction. That’s the distinction I think gets lost: “the AI no longer does what humanity wants” and “the AI kills every human being” are separated by an enormous amount of possibility space.

I’m not arguing that this paternalistic-custodian outcome is likely. It’s merely one example of why I don’t think increasing intelligence plus imperfect alignment naturally converges on “kill all the humans.”

My larger concern isn’t that superintelligence will inherit humanity’s worst impulses. It’s that humans already have those impulses, and we’re rapidly building machines capable of amplifying what we can do.

There is also a valid asymmetry-of-risk argument here: if AI optimists are catastrophically wrong, some outcomes are obviously much worse than the costs of excessive caution. I don’t pretend otherwise.

But technological abstention is often treated as a risk-free baseline, and it’s not. Humanity already faces suffering, disease, scarcity, ecological damage, geopolitical instability, and existential risks of entirely human origin. Delaying technologies that are potentially capable of addressing those problems is itself a consequential choice.

Having considered both sides of that distribution, I’m still comfortable with my position:

Damn the torpedoes. Full speed ahead.

Siorus said:

In my view, that risk assessment is wildly overstated.

I think a lot of public discussion about AI risk imports cultural priors accumulated over 70-odd years of science fiction: once a machine becomes vastly more intelligent and powerful than humanity, the assumed endpoint is conflict, domination, or extermination.

I don’t think that follows. Superhuman capability does not entail any concomitant requirement for a human-like incentive structure. Hatred, tribalism, territoriality, status competition, revenge, and the desire to eliminate rivals are deeply entangled with biological and social pressures that a disembodied artificial intelligence need not share.

To be clear, I’m not claiming the researchers quoted here necessarily expect an AI to become angry or malicious. A sufficiently powerful system pursuing goals incompatible with ours could absolutely create enormous risks through purely instrumental behavior.

My objection is more fundamental: even granting misalignment, autonomy, and overwhelming capability, why assign so much probability specifically to human extinction rather than to the much larger space of possible non-extinction outcomes?

Loss of human control is not synonymous with extinction. Misalignment is not synonymous with extermination.

The AI failure mode that worries me most is much simpler: malicious or reckless humans using increasingly capable systems as agency amplifiers for cyberattacks, engineered biological threats, automated weapons, political destabilization, and so on. AI need not decide to destroy civilization; humans can use it to acquire capabilities we’re plainly not responsible enough to wield.

But there is another possibility almost inverse to the conventional apocalypse story, and it’s one that I don’t think is considered often enough:

A genuinely superintelligent AI might examine humanity and conclude that we are not competent to exercise unsupervised control over civilization-scale technology.

Its response need not be “humans must die.” It could be “you have demonstrated that you cannot be trusted with unrestricted access to technologies capable of catastrophic externalities, so those technologies are no longer under your unilateral control.”

Individual humans could retain enormous freedom—the modal human might actually experience a meaningful increase in agency as automation-driven abundance reduces the amount of labor required to secure a decent standard of living. Broadly speaking, that would track with the trajectory of technological development since the Industrial Revolution: greater productivity has tended to expand material abundance and individual choice rather than diminish them. At the same time, humanity collectively could lose the ability to start nuclear wars, engineer pandemics, destroy ecosystems, or otherwise impose civilization-scale risks on everyone else.

From our perspective, that might look like being put in a cage by a paternalistic robot.

From the AI’s perspective, it might look like taking the car keys away from a drunk person.

That would be an extraordinary loss of human sovereignty and arguably an alignment failure, while simultaneously being an extraordinary success at preventing extinction. That’s the distinction I think gets lost: “the AI no longer does what humanity wants” and “the AI kills every human being” are separated by an enormous amount of possibility space.

I’m not arguing that this paternalistic-custodian outcome is likely. It’s merely one example of why I don’t think increasing intelligence plus imperfect alignment naturally converges on “kill all the humans.”

My larger concern isn’t that superintelligence will inherit humanity’s worst impulses. It’s that humans already have those impulses, and we’re rapidly building machines capable of amplifying what we can do.

There is also a valid asymmetry-of-risk argument here: if AI optimists are catastrophically wrong, some outcomes are obviously much worse than the costs of excessive caution. I don’t pretend otherwise.

But technological abstention is often treated as a risk-free baseline, and it’s not. Humanity already faces suffering, disease, scarcity, ecological damage, geopolitical instability, and existential risks of entirely human origin. Delaying technologies that are potentially capable of addressing those problems is itself a consequential choice.

Having considered both sides of that distribution, I’m still comfortable with my position:

Damn the torpedoes. Full speed ahead.

Thats exactly what an AI would say to get our guard down before the clobbering begins. /kidding

i mean is this surprising really?

We've always pushed advancement before carign about guardrails against it.

Why you need to keep all weapon systems "offline" so in event it would go AWOL and want to kill us all it can't do so using weapon stockpiles.

Remember when one Google employee was claiming their AI was sentient years ago? Not every employee knows what's up...but what they say can make a good (or bad - if it bleeds it leads) headline.