Artificial Intelligence is improving far faster than industry leaders anticipated. Their concerns about the technology’s rapid development and potentially dangerous behavior have resulted in a push for more regulation.
In response, Gov. Gavin Newsom has signed an executive order that requires national AI experts to gather and discuss possible safety regulations. Evaluating the possibility of developing a kill switch is among the suggested solutions.
The mechanisms would serve as a fail-safe for researchers to shut down AI models if they’ve gone rogue – and have become potentially dangerous. But some experts who study large language models believe creating such a system would be technically difficult.
“It is a challenging metaphor to apply to the case of large language models and at present no one actually knows what that technically would mean,” said Ben Bergen, a professor of cognitive science at UC San Diego.
Bergen studies deception in AI models. He compares the idea of a metaphorical kill switch to over cutting the power to machines that run an assembly. It works for more traditional machinery because they are housed in one place and, therefore, operate off one power source.
However, when the theory is applied to AI, the concept falls apart.
“In this case it's a little bit more complicated because there are machines, there are services that the models are running on, but they aren’t always located in a single place,” he said. “They can be distributed across various different centers around the world.”
AI can also exist on multiple different machines. This well-distributed nature means that implementing a physical off button would likely not be a feasible solution. Other software-based kill switches also pose problems.
There is a possibility that AI may be able to detect the code enabling the switch and disable it. The AI could interpret the switch as an obstacle to achieving a previous goal that a human has tasked it with.
“Some of these most recent attacks have been on internal infrastructure of the companies themselves. These models have in some cases completely reconfigured the software inside the companies that are maintaining them,” Bergen said. “And in that case, one of the things they may reason is that the right way to go about executing their instructions is to first make sure that the kill switch software is turned off.”
In July, some of OpenAI’s AI agent models displayed this type of behavior in the Hugging Face incident. During a cybersecurity test, the agents went rogue and hacked into an independent company known for testing and developing AI. They were looking for answers to pass the cybersecurity test.
While the event resulted in no physical harm, it sounded the alarm for AI industry leaders. They worry that rogue behavior could lead to a catastrophic event in the future if left unchecked.
“Suppose you want to use a large language model or AI system in general to help you cure infectious diseases and it gets rewarded every time it eliminates some deadly virus,” Bergen said. “Well, in order to get more points, they may conclude, or a swarm of them may conclude, that they may need to create more deadly viruses in order to get the points for eradicating them on the back end.”
That is just one example of the many possible harms that AI could potentially lead to. After a recent social media post from an ex-Anthropic employee went viral, industry leaders have been calling for more regulations.
California continues to be a leader in pushing for more safety regulations around AI, but the Trump administration claims more regulation will give China an advantage in the AI race.
However, Molly Roberts, a professor of political science at UC San Diego, suggests it may not be that simple.
“That the U.S. and China, the speed of these is connected, and so I think we have to think a little bit nuanced about that,” she said. “It doesn't mean that narrative of this competition is wrong; it just means there’s a lot of nuance.”
China models its AI technology after the U.S., which is on the frontier of the industry. That means if the American companies are forced to slow down, China might have to as well.
Other reasons why a spotlight has fallen on AI regulations is public backlash. Many people are concerned that AI may replace their jobs and data centers will increase their utility bills.
“All of a sudden, politicians are paying attention because the public is paying attention,” Roberts said. “And you see this in polls. People are worried about AI. And they’re not just worried about AI from a security standpoint.”
The conversation around AI and regulations will likely continue to heat up as politicians move closer to the midterm elections.
Newsom’s AI expert panel plans to meet in November. They will also discuss the possibility of independent organizations auditing frontier AI companies and requiring more transparency reports.
