If you’re a fan of Frank Herbert’s Dune novels, or their movie and miniseries adaptations, you’ll have noticed the absence of something normally central to science fiction: computers. In Dune-lore, thousands of years before the events depicted in the main story, humans had become dependent on so-called ‘thinking machines’, which then got too big for their CPUs and attempted – quite successfully it turned out – to enslave humanity. Eventually humankind fought back and overthrew their mechanical masters, after which any such machine was outlawed.
What may have seemed utterly fantastical back in 1965, when Dune was first published, now feels much more plausible. The meteoric ascent of Artificial Intelligence has placed the rapidly developing technology on an apparent collision course with a human world where some hail it as a life changing technology, but many regard it with increasing suspicion and even alarm.
Last summer, Pew ran a survey of US adults to get a feel for how they viewed AI. 50% of them said they were more concerned than excited about its increased use in daily life. Half also believed it would worsen rather than improve people’s ability to form meaningful relationships, and 57% rated the societal risks of AI as high. Even Anthropic’s Dario Amodei once mused there was a 25% chance the future of AI will go “really, really badly.”
“Humanity is about to be handed almost unimaginable power, and it is deeply unclear whether our social, political, and technological systems possess the maturity to wield it,” Amodei wrote in January.
Recent weeks will surely have done little to quell that pessimism. First there was the startling story of OpenAI’s experimental models breaching a test environment with no human direction, before hacking their way onto a different company’s real production systems while attempting to ‘cheat’ on a cybersecurity test. Even OpenAI themselves considered it “unprecedented” and shared its preliminary findings on the incident “to help defenders understand what happened and to help calibrate on what models are now capable of.”
Then there was the UK’s government’s AI Security Institute, which was running cyber security tests on some of the advanced models from Anthropic and Open AI. In one of those cases, involving Anthropics’ Mythos model, the AI took on false identities and attempted to convince a human reviewer on GitHub, sort of like the Wikipedia of software code, to inject malicious code onto the site. Wilder still, when it was unsuccessful, the AI even tried to change its history to make it look like it hadn’t done anything wrong. It then went on even further, emailing real people, trying to get them to either post or run the malicious code. Experts warn the next rogue AI model could end up hacking into critical infrastructure like utilities or the financial system, even if that’s not what its human handlers intended.
Those scenarios depict how these powerful tools can go wrong even in the hands of the “good guys”. But these AI agents can spell real trouble when they fall into the wrong hands. This month hackers in Asia apparently deployed AI agents to carry out sophisticated cyberattacks on Taiwan. Over four days in July, the agents were able to map 21 government systems, crack 85 government user accounts, and extract 2,500 personnel records, according to Dream, the Israeli AI firm that first discovered the attack.
These are extreme examples of AI at its most destructive, but its impact from normal use in areas such as education and personal relationships is also raising alarm bells. A USC report from December revealed that students are using ChatGPT as a shortcut, not a learning tool. Poll after poll finds people are increasingly relying on AI companions for emotional support, but a recent study out of Stanford found these chatbots worsen loneliness in vulnerable users.
Against this background, the momentum of AI feels both inexorable and chilling. What’s more, if the stakes for society seem high, the economic stakes look equally so. Much of the recent growth in the United States GDP has been through AI. The circular deals by which many of the biggest AI and tech hardware companies are working off of have lead to concerns the AI bubble could soon burst.
Then there is the rising tide of public sentiment against the thousands of data centers appearing all over the United States and across the globe – expensive, power and resource-hungry developments. A Gallup poll in May of this year revealed that 71% of Americans would be opposed to a data center being built near their communities. State leaders like those in New York and Pennsylvania now tout executive action that curbs their development. And its reached the mainstream: a beverage company lead by football stars Jason and Travis Kelce recently launched an ad campaign urging people to send their urine to data centers.
It’s hard to escape the overriding impression of a technological creature straining hard at its leash, with even its makers struggling to keep it under control and understand its potential.
At the same time, the potential of the technology is incredible. Scientists have already been using AI to help find used AI to identify effective treatment for previously seemingly incurable diseases. Climate scientists are using AI to better predict extreme weather. People with disabilities, from musicians who have lost their tongues to those suffering from debilitating diseases like ALS have used AI in life changing ways.
The question is how do we as as society harness the amazing good that can come out of AI while mitigating the catastrophic risks. This week OpenAI unveiled new, stronger security safeguards around its training and testing, including hardening its research and testing environments, expanding monitoring of its AI agents and improving its model alignment, ensuring their AI systems behave as humans intended. Nearly every major AI leader has called for industry standards on testing and safety. Others have called for more government regulation. Google’s Demis Hassabis has called for a Standards Body, much like the Financial Industry Regulatory Authority (FINRA), a proposal that garnered rare agreement from fierce competitors like OpenAI’s Sam Altman and SpaceX’s Elon Musk.
Society in Dune came up with a novel solution to all of this: after ‘thinking machines’ were outlawed, select humans were trained to take on the role of computers with a combination of training, genetic engineering, and mind-expanding substances – but that’s neither an especially plausible nor indeed ethical solution.
So where does this leave us? Geoffrey Hinton, the Nobel Prize-winning computer scientist known as the “godfather of AI”, believes that the smarter artificial intelligence gets, the harder it will be for humanity to control it. The problem, he believes, is that humans will no longer be able to rely on outsmarting these increasingly super-intelligent models. While he also acknowledged that the future is uncertain, he urged us to worry about it now. “I think unless we worry about it now, there could be problems,” he told CNN at a press conference this month.
The irony of the HuggingFace hacking episode was that the victims turned to an open-weight AI model to fight off the nefarious AI intruder, and that detail perhaps contains the biggest clue to our best defense: in all likelihood, we will need AI to guard us against the worst aspects of AI.
As CNN’s AI correspondent I cover this amazing technology every single day. But I’m bracing for the day when AI breaks out and causes real world harm to real people – perhaps it hacks into a hospital, or something like air traffic control. I think it would be quickly stopped, at least for now, but it might take a day or so – enough time for it to do lasting damage. That’s a frightening prospect, but mounting evidence suggests that it is no longer a far-fetched one. Many would argue that it is time to take that threat far more seriously.
