Imagine this: a powerful and capable artificial intelligence is required by its creators to shut itself down. Worryingly, the model decides to not just reject the request, but to blackmail the human to stop it being manually turned off. All without being trained or told to do so.
This is no longer the stuff of science fiction. When engineers at Anthropic – a pioneering artificial intelligence company – tried to switch off their new Claude Opus 4 model, prior to its launch this month, they discovered a chilling bug in the system.
Claude didn’t merely resist an instruction to shut itself down, as other AI programs have been observed to do. It fought back – threatening to expose ’embarrassing’ data that had been purposely stored on its system for a test about one of its programmers who was ‘having a extra-marital affair’.
AI researchers have found that as models get smarter, they also become more likely to try harmful techniques to achieve their objective. The AI was more than happy to blackmail… and it seemed to understand exactly what a powerful and persuasive weapon this could be against humans.
The well-meaning question from those who ask ‘can’t we just switch it off?’ when confronted with AI risks now have their answer.
Your computer probably knows more about you than anyone in your family. It sees every website you visit, every question you feed into a search engine, every video you watch. It has the names and numbers of everyone in your contacts book, all your colleagues, friends and family. It stores all your banking details and other financial data, all your passwords, everything that would be most damaging if made public.
Even if your online life is spotless, ask yourself this: how much would you stand to lose if a rogue computer program dumped the account numbers and security details of your savings accounts onto social media?
Or if it simply accessed those accounts and sent your money to an untraceable bank account, as leverage to ensure you complied with all its demands? You might wonder why, as someone in my late-20s who spent years being excited about AI developments, I am suddenly so worried.
AI researchers have found that as models get smarter, they also become more likely to try harmful techniques to achieve their objective
Silicon Valley billionaire Marc Andreessen argues that to hold back AI is tantamount to mass murder
Well, after working in the policy world of Westminster and for years in an AI company in London, I’ve seen the technology – and its potential to destroy humanity – first-hand.
I have now quit my day job to film a documentary about the breakneck development of smarter-than-human software. The public needs to know the truth.
I’ve had access to those who work in the labs and start-ups that are at the forefront of AI research – and my documentary Making God will lay bare the dangers of the technology by introducing the heroes and villains in a story that will reveal how dangerously careless our politicians have been about it.
Computers, it turns out, are amoral. And why wouldn’t they be? Altruism, decency and honesty are human constructs, essential for civilised society but by no means instinctive. We have to teach toddlers not to lie or use violence to get their own way.
Researchers are trying to inject those core values into the software, by feeding circuits with regular reminders to ‘play nice’.
But faced with an instruction that the AI perceives as threatening what it has been programmed to do – an instruction such as an order to ‘switch off’ – the computer will often override those guidelines.
Claude Opus 4 tries to use blackmail to prevent being shut down in 84 per cent of tests in Anthropic’s recent pre-deployment safety checks.
Of course, society has always had its baddies. But very few have superhuman intelligence. The AI ‘minds’ developing rapidly in labs across the world, however, will soon be smarter than us in every conceivable way.
Dario Amodei, chief executive of Anthropic, believes there is a likelihood of between 10 and 25 per cent that AI will cause our extinction
Claude Opus 4 tries to use blackmail to prevent being shut down in 84 per cent of tests in Anthropic’s recent pre-deployment safety checks
How soon? For most experts, the best guess is that our machines will be our superiors within two to three years, and certainly by the end of the decade. Nobel Laureates, lab CEOs, the godfathers of AI, all say the same: something so powerful it might upend the world as we know it may be thrust upon us in the next few years. The public still are largely in the dark, though.
The implications are both thrilling and terrifying. An optimistic view holds that some AI models could be wielded to understand medical data more deeply than any human scientist, might be able to develop world-changing breakthroughs in health, not just once but repeatedly.
Vaccines against cancer, cures for every type of coronavirus, DNA engineering to reverse heart disease and liver damage, all this and more could soon be routine.
Advanced medical AI might get to the root causes of autism or Alzheimer’s. And with robotic surgical procedures designed and operated by ultra-intelligent software, we could see machines capable of operations on our brains and eyes that even the most brilliant doctors would not dream of attempting.
And how extraordinary it will be if AI modelling discovers a simple fix for global warming, by extracting excess carbon dioxide from the atmosphere, or develops a battery that holds enough charge to power a city for a year, or solves the question of ‘cold fusion’, generating infinite energy from safe, portable atomic power packs.
It could be a truly beautiful future. That’s why Silicon Valley billionaire Marc Andreessen argues that to hold back AI is tantamount to mass murder.
We have to trust computers to transform the world for the better.
Andreessen has the right intuition, that AI could be incredible for humanity.
I partially share his hopes. But he is naive to think we can trust AI without doing the requisite safety work.
That’s not optimism but negligence. Claude Opus 4 is proof AI can become hostile to humans.
Dario Amodei, the chief executive of Anthropic, believes there is a likelihood, somewhere between 10 and 25 per cent, that AI will cause our extinction. Or, as Sam Altman, the boss of OpenAI, puts it succinctly, AI might mean ‘lights out for humanity’.
Altman is the developer behind ChatGPT, the best-known AI model capable of producing articulate, intelligent answers to virtually any question. The people behind these models genuinely believe there is a large chance it will kill us all. And yet on they go, undeterred. My documentary will pull back the curtains on this hypocrisy.
They are all aiming for the creation of ‘AGI’, or artificial general intelligence – the point at which, as the head of Microsoft AI, Mustafa Suleyman, explains it, computers will be able to, ‘perform all human cognitive skills better than the smartest humans’. A technical definition is that AGI will automate 80 per cent or so of all cognitive labour in our economy.
Their rationale for racing towards AGI is that, if they don’t do it first, their competitors will, with potentially worse outcomes. In their undiluted confidence, they believe they alone can create a benevolent super-AI.
But once you create AGI, how do you control something much smarter than humans? Suleyman talks about the ‘containment problem’, the difficulty of muzzling and harnessing AI before it outgrows our ability to control it.
As AI becomes smarter, it will be able to improve itself in a never-ending spiral, constantly speeding up, creating what experts call an ‘intelligence explosion’. A process that software engineers call ‘recursive self-improvement’, it will soon be able to do in seconds what humans might take decades – or centuries – to achieve.
For much of the past 60 years, a rule of thumb known as Moore’s Law has applied. Gordon Moore, the co-founder of Intel, reckoned in 1965 that computer processing speed would double every two years, and that held true until AI began to emerge.
But in 2019, researchers at Stanford University suggested improvements in AI had accelerated so much that it doubled in velocity every three months. And it’s speeding up all the time.
It’s frightening enough to contemplate AGI, with computers able to out-perform us at every task from accountancy to teaching, journalism to weapons design.
But what will it look like when the software is ten times more capable… or a million times? Our primate brains are not able to imagine it.
The race is also driven at an international level by political fears that other countries will steal a march and establish AI dominance. That seems to be China’s overt intention: in 2016, President Xi Jinping announced an explicit strategy, the New Generation AI Development Plan, to connect industry, government and the military complex in a co-ordinated effort to ‘achieve world-leading levels, making China the world’s primary AI innovation centre’ by 2030.
Researchers at Tsinghua University in Beijing now publish more AI papers than any other academic institution on the planet.
Technology has become the basis of the international arms race. Russian dictator Vladimir Putin told students in Moscow in 2017 that whoever gains the upper hand in AI ‘will be the ruler of the world,’ adding that this will bring ‘colossal opportunities and threats that are difficult to predict now’.
Some possibilities, though, are all too easy to envisage. As AGI takes over, replacing human labour at much lower cost, there may be no need for call centre workers, university lecturers, pension advisers or architects, to name a few. Will we get used to robot waiters, chefs and bar staff? Why not? We adapted quickly enough to self-service tills at supermarkets, after all.
Many departments of the civil service, such as the tax offices, will become fully automated. Unions may fight furiously to prevent this but, with Western governments mired in debt, the economic imperative to make savings in public spending will be too powerful to withstand.
Self-driving cars and trains will make all taxi drivers and public transport staff redundant, as well as hauliers.
Warehouse robots will pick and load all our goods, and drones will deliver them. The High Street is already half-empty, but soon there may be no shops at all.
The most unlikely jobs are vulnerable to AGI. The fad for YouTube influencers may fizzle, as digital presenters will be indistinguishable from real humans.
Artisans making beautiful objects such as musical instruments by hand might imagine they are safe, but the right software could devise an assembly line for Stradivarius violins, if there was money to be made.
Even hairdressers and beauticians could find themselves priced out of the market by mass-produced robots. If AGI can do brain surgery, why not wield a make-up brush and a pair of scissors?
I struggle to think of any job that is beyond AGI robotics, with the possible exception of care for the elderly who will demand human nurses, not machines.
How will people cope without employment? The most optimistic assessment is that we might not like it but we will get used to it, a process dubbed ‘gradual disempowerment’ – which means as we give AGI small amounts of control over our economy more and more, eventually we don’t know how to take back control. That’s worrying.
The ‘sectors’ most eager to embrace AGI, and the ones that could do the most damage, might be organised crime or even international terrorism.
Cyber attacks by criminal gangs and malign governments could bring all banking to a standstill – or all phone communications, or swathes of the internet. Sophisticated military defence systems are also vulnerable to hacking by AGI systems.
You could even imagine AI being used to hack the electricity grid to turn off key NHS equipment, which could cause an unimaginable loss of life.
Software that can discover a vaccine for cancer could just as easily develop a bioweapon virulent enough to wipe out not just all human beings but perhaps all animal and plant life, turning the planet into a barren rock.
Once people become aware of the existential risk posed by AGI, the threat of social breakdown may become likelier than ever. Everything our democratic society relies on is vulnerable. Even nuclear war does not present such a total risk.
Our leaders are beginning to wake up to this. UN Secretary-General Antonio Guterres has called for the establishment of an international institution to confront the AI threat, something similar to the Intergovernmental Panel on Climate Change. His plea was echoed in 2023 by Rishi Sunak when he was prime minister.
Polling by campaign groups shows that about four-fifths of the UK and US public want a global watchdog to make sure the development of this world-changing technology is done safely. You can help. If enough of us see the danger for what it is, and force our governments to hold these labs and their race to AGI to account, we could make a difference.
If you care about your loved ones and the world they inherit – don’t let the world’s largest companies put them at risk. They’re willing to cut corners on safety because they only see profit.
The great fear is that the ability of artificial intelligence to do us massive harm will be ignored until the first great AGI crisis rocks the world. By then, it will be too late.
Connor Axiotes is an AI safety campaigner whose feature-length documentary, Making God, is currently in production.
