This story was originally published by CalMatters. Sign up for their newsletters.
Gov. Gavin Newsom announced Friday that he is directing state agencies to develop artificial intelligence safety recommendations, some of which were part of a bill he vetoed two years ago.
His change of tune on those measures, including requiring AI developers to include a “kill switch” to shut off certain programs in emergencies, comes in response to growing public alarm over the fast-developing technology’s potential “catastrophic” harms such as AI creating a bioweapon or bringing down the power grid.
Newsom is seeking recommendations from the Government Operations Agency and the Governor’s Office of Emergency Services by Nov. 16, meaning they could form the basis of a special legislative session on the issue. He floated the possibility of calling one in Politico on Thursday.
Newsom has sought in recent weeks to respond to several incidents in which AI agents from leading developers OpenAI and Anthropic broke away from testing environments, gained access to the open internet and carried out cyberattacks on other websites and companies. Earlier this month, a researcher set off a panic when he resigned from Anthropic while warning the technology could “kill all humans;” the researcher’s former colleagues were quick to agree.
The hackings have captivated the public and seem to confirm AI researchers’ fears that the fast-developing technology is improving its own capabilities beyond human control. Several industry leaders last week endorsed Anthropic CEO Dario Amodei’s call for a slowdown in development.
Newsom’s order directs the state agencies to recommend changes in state law to require external safety evaluators at AI companies, as well as require independent verification for AI companies’ safety frameworks that are required under current state law. He’s also seeking recommendations on how to require a kill switch, and how to expand the safety incidents companies must report to the state to include “a range of loss-of-control incidents” such as the recently reported hackings.
Regulation is unlikely from the federal government; President Donald Trump remains all-in on speeding up AI development in the face of competition from China and said the calls for a slowdown are a “conspiracy.”
Newsom, a likely 2028 presidential contender, said Friday in a press release that should “alarm every American.”
“We’re going to speed up our work on substantial and responsible AI oversight before it's too late,” he said.
California under Newsom has enacted several first-in-the-nation laws to curb the potentially existential risks of AI. In 2025, he signed Senate Bill 53, which requires large AI model developers to publish frameworks on how they manage safety incidents and assess catastrophic risk — defined as incidents involving 50 or more deaths, chemical or biological weapons or more than $1 billion of theft or damage. They also must report critical safety incidents to the state. Illinois and New York have similar laws.
That law sets a high bar for what incidents must be reported, though, including only those involving deaths, injuries or the “materialization of a catastrophic risk.” Advocates for greater tech regulation want the threshold to be lower to cover more incidents like the recent hackings.
The measure was a compromise — a requirement for transparency rather than regulatory control — after Newsom in 2024 vetoed Senate Bill 1047.
The vetoed measure would have required the largest AI developers to receive safety auditing from third-party, external monitors, develop a kill switch and be subject to clearer legal liability if their technology causes harm. The measure divided Silicon Valley and was fiercely opposed by several of the largest AI developers.
The measure was a compromise — a requirement for transparency rather than regulatory control — after Newsom in 2024 vetoed Senate Bill 1047.
Both bills were authored by Democratic Sen. Scott Wiener, whose San Francisco district is home to the most prominent AI companies.
“We must act with all possible haste to address the serious risks of AI-driven catastrophe, and I commend the governor for taking this important step,” he said. “I’m glad to see the strong AI safety framework I laid out in 2024 is still being used to guide policy in 2026.”
The vetoed measure would have required the largest AI developers to receive safety auditing from third-party, external monitors, develop a kill switch and be subject to clearer legal liability if their technology causes harm. The measure divided Silicon Valley and received fierce opposition from several, but not all, of the largest AI developers.
Newsom wrote two years ago in a veto message that while he agreed with the need to require safety protocols, the bill could restrict AI development at large companies “at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.”
“I agree with the author — we cannot afford to wait for a major catastrophe to occur before taking action to protect the public,” he added.
Since then, more in the industry have embraced third-party evaluations. This year, Newsom signed two laws to verify, register and set standards for those external monitors. But the new laws do not require companies to use them.
In his recent call for a slowdown in AI development, Amodei also called for third-party monitoring; both Anthropic and OpenAI say they will commit to the practice.
This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.
