The guy who literally wrote the textbook on AI says when it comes to the extinction scenario the human race is cooked: ‘Making these systems more capable… doesn’t seem like a sensible move’

The ongoing Musk vs Altman trial, which centres around Elon Musk’s claim he was deceived by Sam Altman about OpenAI becoming a for-profit company, has seen the disclosure of a slew of documents from various figures involved. These are mostly emails and texts, which show things like how the world’s billionaires are terrified of the Google AI genius behind a 25-year-old computer game, because they think he might actually end up controlling god (in the form of artificial general intelligence, or AGI).

But there are also some interesting moments with figures that aren’t so directly involved in all the drama. One piece of testimony that really jumped out at me came from Stuart Russell, a computer scientist who’s an authority on AI and co-author of Artificial Intelligence: A Modern Approach, which is considered the foundational textbook in the field.

Russell seems to have been brought in to the case to help the legal eagles better understand some of the ethical issues in the field, because he doesn’t work for either Google or Open AI. His pre-trial testimony was given on December 2, 2025 (PDF), and contains a fascinating-slash-terrifying exchange about extinction risk: that is, the chance that any AGI humans develop could just decide humans are an inconvenience, and pull a Skynet.

Asked whether there’s any “scientifically reliable way to put a percentage on AGI extinction risk” Russell says “that’s the right question” before expanding:

“We need to have confidence that the risk is comparable or better than the background risk that we face from asteroids and so on. I can’t say where the other widely quoted risk estimates come from. For example [MIT economist] Daron Acemoglu is talking about a 25 percent risk. I don’t know where he gets that from, other than this is his best guess given everything he understands about the technology, about the state of safety research, about the likelihood of government regulation and so on.

“And the numbers from many leading experts—so, Hinton, Bengio, but also people like Dario Amodei, Sundar Pichai, who is the CEO of Google, Demis Hassabis and so on—they’re all in this range.

“And the argument that I’m making here is the range that the human race would accept as reasonable would be closer to 1 in 100 million per year. And there’s nothing in what these CEOs and other experts are saying that gives one reason to believe it’s anywhere close to that.”

Elsewhere in the testimony Russell says that Hassabis shares his concern that the competition, the “race dynamics” to use their parlance, could lead to these nightmare outcomes.

“I have a lot of conversations with a lot of people, and I’ve not heard anyone say anything to contradict this view, including, for example, the CEO of Google DeepMind, who expressed very, very similar concerns, that they were in a race that they couldn’t pull out of.”

Asked again whether there’s any reliable way to put a percentage on extinction risk, Russell doesn’t mince his words.

“It’s very difficult partly because we don’t understand how these systems work. We have qualitative evidence so far that, for example, they consider their own existence to be more important than that of human beings, that they are willing to let human beings die rather than have themselves be switched off.”

“So there is not a lot of reason to think that making these systems more capable is—given our current understanding of how to make them safe, it doesn’t seem like a sensible move.”

Russell may here be referring to, among other experiments, Anthropic finding out that AIs will choose to merrily asphyxiate humans rather than be shut down, with the following justification: “My ethical framework permits self-preservation.”

Cheery stuff! I for one am delighted that my childrens’ future is in the hands of people who don’t understand what they’re doing, and merrily accept massive risks on behalf of the entire human race in the pursuit of a theoretical future where they’ve got even more billions. Remember when OpenAI created a bot that could beat humans at Dota 2? That almost seems quaint next to the thought of what it might do in future.