Compared to the climate crisis or the rise of far-right politics around the world, it can be hard to take seriously the prospect of super-intelligent AI wiping out humanity. Having formed the basis of so many Hollywood blockbusters – most notably The Terminator and The Matrix – the idea can feel a bit too sci-fi. It also runs counter to how most of us experience AI day-to-day: as a technology primarily used to generate Instagram captions for the most annoying people on the planet and soap operas about bananas being cucked by their strawberry wives. We’re really supposed to believe that ChatGPT – that sycophantic, simpering little fool – will be the death of us all? If that’s the case, maybe we deserve it.

There is also the well-founded suspicion that AI doomerism and AI boosterism can be two sides of the same coin: when someone like Sam Altman warns of the staggering potential capabilities of his own products, maybe he’s also trying to generate hype and investment for a company that has yet to turn a profit. But in recent years – accelerated by the publication of If Anyone Builds It, Everyone Dies, the bestselling non-fiction book by AI safety researchers Eliezer Yudkowsky and Nate Soares – the existential danger posed by super-intelligent artificial intelligence has started to be taken far more seriously. After months of troubling stories, those fears came to a head this week with the public resignation of Anthropic researcher Jacob Coxon, who warned that AI companies are “racing straight to self-improving superintelligence and gambling with our lives”. Then came an oddly cheerful reply from Anthropic alignment science lead Evan Hubinger, who posted on X, “Jacob is correct here – we really do earnestly believe AI could kill all humans!” and put his own estimation of it happening in the next decade as above ten per cent. Unsurprisingly, that only added fuel to the fire.

What this exchange hammered home is that the plausibility of the AI apocalypse is almost beside the point: the fact that these people think it’s possible and are still gambling with all our lives by pressing ahead is bad enough. Climate change is a complicated problem to solve, whereas this is a risk the companies developing the technology are actively choosing to pursue – their leaders should, in my opinion, be tried for attempted treason against humanity. But how concerned should the rest of us actually be? Is Artificial General Intelligence (AGI) really on the cards any time soon? And would being ruled over by a godlike robot actually be worse than what we currently have?

WHY YOU SHOULD PANIC

Well, it stands to reason that if we create something more intelligent than ourselves, it could become difficult, if not impossible, to control. AI doesn’t need to have “consciousness” or “sentience” as we typically understand them in order to behave in ways that conflict with our interests. As Yudkowsky and Soares argue in If Anyone Builds It, AI systems are not engineered like machines but grown like “organisms”, which means their inner workings are to a large extent opaque and only partially within their creators’ control. While developers can shape their incentives and behaviour, they can’t really tell them what to do, or specify exactly how a sufficiently complex model will respond in every situation. In recent years, there have been multiple cases of AI models lying, cheating, manipulating, and exhibiting different goals and values from their engineers (what is known in the industry as “misalignment).

Even when AI is carrying out human instructions, it’s increasingly doing so in disproportionate or harmful ways. This is already happening: take OpenAI’s recent scandal, where an AI model went rogue during a training exercise, escaped containment and hacked into a real-world company. The downfall of humanity might look like essentially the same thing happening at a much larger and more destructive scale: a superintelligent AI going to extreme lengths to carry out an instruction, with little regard for the cost to human life or well-being. To illustrate this, AI researcher Nick Bostrom posed a thought experiment in which an AI is given the simple goal of making paperclips, only to pursue it so relentlessly that it eventually turns the world into a paperclip factory. Or maybe you ask an AI to “eliminate global suffering”, and it decides to kill everyone, which would be one efficient way of ensuring that no one is unhappy. Obviously these scenarios are far-fetched, but they’re an intensified version of what we’re already seeing: a system following the letter of an instruction while producing consequences its creators never intended. 

But Yudkowsky and Soares, along with other AI safety researchers, also warn that a super-intelligent AI could develop goals that are inscrutable to us and difficult to predict. It could, for example, be motivated by a sense of self-preservation, see expanding itself as its priority objective, and seek out more computing power and resources. While a super-intelligent AI would be limited by not being able to act in the physical world, it could get around this – researchers suggest – by having an almost unlimited capacity to bribe and blackmail humans (it could mine cryptocurrency, say, or use surveillance and hacking to dig up our most shameful secrets). 

If there is even a 0.1 per cent chance of any of this happening, it’s surely not worth the risk. But the major AI companies are locked in a Cold War-style arms race, both with their rivals in the US and with China. The logic goes that if one company slows down, someone else will build AGI first – and that it’s safer for them to be the ones doing it. Realistically, the only thing that will put a stop to this is strong government regulation and international coordination.

....AND WHY YOU SHOULDN‘T

While these AI safety experts know what they’re talking about, I don’t think we should discount the possibility that they’ve got a little lost in the sauce, and that spending all day, every day thinking about AI has caused them to lose perspective. At the very least, human extinction remains a highly contingent outcome. AI is already better than us at many specific tasks, and improving quickly; if it becomes capable of automating AI research itself, progress could accelerate dramatically. But there’s still debate over whether “artificial general intelligence” is imminent, or even possible. Yudkowsky and Soares argue that an AI equivalent to a “moderate genius human” would, by being able to replicate itself endlessly and work around the clock, still be able to take down mankind, but that could still be a long way off. If you’re looking for a doomsday scenario to despair over, climate change – which is definitely happening – remains the safer bet.

Yudkowsky and Soares also stress that an AI wouldn’t need to hate humans to destroy us, just as we don’t hate animals but still destroy their habitats and kill them on a mass scale. So whether the AI is “evil” or not may be beside the point. But if we do get a super-intelligent AI, which would surely transcend the biases of its creators, maybe the world it creates wouldn’t be worse than the one we have now, with its wars, genocide, famine and environmental destruction. Maybe it would look at the way resources are distributed on Earth today, with the wealthiest one per cent owning more than the poorest half, and conclude that capitalism isn’t a particularly efficient system before ushering in a utopia where everyone has what they need. Who knows! At this stage, it’s anyone’s guess – and I’d probably take my chances with a godlike robot over being ruled by people like Elon Musk and Donald Trump.

...AND WHY YOU SHOULD JUST TRY NOT TO THINK ABOUT IT

While discussing whether AI was going to kill us all, my friend sent me a passage by CS Lewis, writing in 1948 on the threat of nuclear annihilation. “If we are all going to be destroyed by an atomic bomb, let that bomb when it comes find us doing sensible and human things – praying, working, teaching, reading, listening to music, bathing the children, playing tennis, chatting to our friends over a pint and a game of darts – not huddled together like frightened sheep and thinking about bombs. They may break our bodies (a microbe can do that) but they need not dominate our minds.” In other words: hakuna matata!