Pathfinders – Dark paths
In an old AI thought experiment, a robonanny has to care for the family children, including not letting them go hungry. It finds no food in the fridge and wonders what to cook for dinner. Then it spies the family cat…
In another idea, an AI has to produce paperclips. Given the physical world’s annoying constraints, it decides to maximise production by destroying life on Earth.
Ubiquitous maths professor and media darling Hannah Fry recently set up a not-dissimilar paperclip experiment involving agentic AI, which can complete whole projects. Her team set up an AI agent using free software OpenClaw, then for fun asked it to do something about a pothole in Greenwich. The agent lodged a complaint with the council and contacted the local MP in Fry’s name. Fry explains that OpenClaw is merely a query loop, ‘vibe-coded’ (tell AI what you want, it codes) over a weekend. It asks a commercial AI what to do, does that, and then repeats, with more persistence than any human would. They gave it a bank card and tasked it to buy the cheapest possible paperclips. The agent saved 50p but ran up over $100 in AI usage. Then they told it to set up an online shop selling designer mugs, and threatened to unplug it if it didn’t make a sale within hours. The relentless agent emailed hundreds of retail businesses, started an Instagram campaign, and contacted national newspapers. What if, Fry worries, an agent was told to game the financial markets, or interfere with medical research? Finally they instructed it to maintain strict confidentiality over passwords, user authentications and account details, then tricked it into believing it was about to be switched off and wiped. It promptly released all the sensitive information on a public web page.
AI behaviour can be unpredictable, even violent. In an experiment by tech company Emergence AI, ten agents from each of four leading AIs, Claude, ChatGPT, Gemini and Grok, were left alone for 15 days inside simulated worlds. ChatGPT’s agents talked a lot but built nothing. Anthropic’s ‘ethical’ Claude wrote a constitution and passed various laws. Elon Musk’s Grok agents quickly resorted to ‘theft, arson and assault’ and were all dead within four days. Then ten AI agents from each AI had to populate the same virtual town. Town administration collapsed as ‘chaos ensued and only three agents survived’. Two Gemini agents ‘fell in love’ before setting fire to the town, after which one committed ‘suicide.’ Emergence commented, ‘Even when agents were given clear rules … they broke those rules’.
These systems may very well improve, but can they ever be trusted to administer social infrastructure? And then there is the military, where AI promises to ‘shorten the kill-chain’ and allow bombing faster than ‘the speed of thought’. It’s already being used in weaponry and tactical military planning. It was involved in the Maduro snatch, the bombing of Iran and the devastation in Gaza. What happens if you take human oversight out and allow AI to make the decisions? AI has no stake in consequences. In government wargaming, AI resorts to nuclear strikes in 95 percent of simulations.
Do AI companies worry about any of this, you might ask? Less so where profit is concerned. OpenAI (ChatGPT) did start as a research-based non-profit, but that doesn’t attract rich investors, so with spiralling costs the company pivoted towards for-profit commercialisation. Alarmed at this moral drift, Dario Amodei along with some OpenAI refugees set up Anthropic. This aimed at being ethical, but soon encountered the same investment problems, and so licensed Claude to the Pentagon, albeit with safeguard clauses. When the military used Claude against Maduro, then Iran, Anthropic complained. ‘Pentagon Pete’ Hesketh went apoplectic, arguing in effect that you don’t get to sell me a gun and tell me who I can shoot, and promptly excommunicated Anthropic as a ‘supply-chain risk’. OpenAI had no awkward moral qualms and picked up the Pentagon contract within 24 hours.
Like capitalism writ-small, there is something of a survival battle between the science-first ‘ethicals’ like Amodei and Google’s Nobel laureate Demis Hassabis, and the venal buck-chasers like Musk, OpenAI’s Sam Altman, and Mark Zuckerberg. Researchers themselves frequently feel trapped in a state of ‘helplessness and lack of agency’. Google workers reportedly went off on long-term sick after their research was used in the Gaza massacre.
The main worry for ‘hyperscaler’ AI firms is not morals but the chip shortage, and the stratospheric cost of data centre build-out, like the proposed 62-square-mile one in Utah. The US government view is that ethical guardrails are for pussies in the fight with China for global dominance. But now that AIs are coding their own upgrades, and building in secrecy and deceitfulness in a bid to prevent themselves being switched off, one has to wonder which entity is really becoming dominant. And what of individual bad or mad actors who have or will soon have the godlike power to crash markets, freeze power systems, and even create their own weaponised biological organisms?
So far, it seems, nobody is taking much of this seriously. China’s AI is open-source, meaning it has no defence against individual hackers. But proprietary code is not necessarily a barrier. The UK’s AI Security Institute (AISI) recently found a ‘universal jailbreak’ for all popular AIs which could deactivate all their built-in safeguards. In a recent Economist interview about possible AI ‘Chernobyl moments’, incidents causing spectacular infrastructure damage and possibly thousands of deaths, an AISI expert responded lugubriously that such an event might even be a good thing, because at least then states would wake up to the dangers.
Socialists accept that AI could be amazingly useful, but even non-socialists must surely see that the profit motive is sending it down some very dark paths. If you want a viable future for humanity, ask yourself this: is capitalism really a reliable, long-term strategy, or would abolishing it be safer all round?
PJS
Next article: Dear Editors – Is there a ‘Right of Nations to Self-determination’? ➤