In light of events over the last few weeks, I’ve been asked by a number of readers about the chances of AI causing human extinction. While this seems like a question out of some science fiction scenario, it’s now common to see exactly that formulation making it into headlines in all parts of the media eco-system.

To cut to the chase: I would put the chances that AI causes human extinction anytime in the near future (let’s say the next 10 years) at exactly zero. This is not to say that these technologies don’t pose both serious challenges and serious risks for society. It’s just that even talking about extinction frames the debate in a way that is:

a) kind of absurd.

b) takes our eyes off the ball about the real danger of having these flawed technologies shoved down our throats by people who stand to make ungodly amounts of money from them.

Let’s start with part a: Extinction.

No one seems willing to explain exactly how AI will drive the human race extinct. Extinction is, after all, a pretty high bar to achieve. How for example would a rogue AI manage to kill everyone in Paraguay? When pushed we get vague handwaving about making a super-virus or launching all the nuclear weapons. The first makes no sense because AI doesn’t have any … um … hands. Making and releasing a super virus requires labs and distribution which means physical bodies in the physical world. While we see amazing videos of robots running races and doing kung fu, the fact is the world is a long, long way from functioning human-form automatons. The nuclear weapons scenario would require lots and lots more stupid things be done by humans in terms linking nuclear security to AI. And, while I know this is harsh, even a full-scale nuclear war would not exterminate humanity.

So, extinction … I don’t think so.

But what are the real dangers posed by AI?

Fundamentally Claude, Chat-GPT etc. are software products. To the degree that the world depends on software they pose a danger. Given how much the world depends on software we clearly are at risk. Why? Because Claude, Chat-GPT etc. are deeply flawed software products. This is the thing that never gets talked about. OpenAI and Anthropic talk about their programs like they are separate sentient beings which act on their own accord. That’s not the case at all. They are software systems with capabilities the companies don’t understand. If this was some other product behaving as badly - like an aircraft autopilot program -the company making it would go out of business.

And that gets us into new territory.

The frontier model LLM companies are burning through cash like a supernova. They have no “moat” - meaning they are running out of that cash with no way to make it back for the foreseeable future. That doesn’t mean their products don’t pose a real danger. It does mean we should be very suspicious of their intentions. They have not earned our trust.

I think there is real possibility that some agent swarm ends up taking down a city’s water system or an air traffic control grid. It’s like we are all just sitting here waiting for it to happen. And that is why, for the US at least, this next election matters so much. I have been really happy to see public outcry over AI beginning to get serious. It’s been years of these companies telling us that, at best, their products will lead to everyone being fired while, at worst, they will end the world. Now they seem shocked that ordinary folks are saying, “no thanks”.

It’s not that these AI products are super intelligent. They’re not. They are very good at solving some kinds of software-able problems. Other than that, they can be super stupid in ways that can be very dangerous for us. While I don’t think extinction is the problem, there are other problems they could cause that will be serious enough to demand reining in this “race” right now.

=============================================================

PS If you have specific questions or issues you want me to address leave a comment on the website or email me at [email protected]

PPS I could not get this proof-read so please excuse typos etc.

— Adam Frank 🚀