California Senate Bill 1047, formally titled the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, was a proposed state law aimed at regulating the development and deployment of large-scale artificial intelligence systems. Introduced by State Senator Scott Wiener in February 2024, the bill sought to impose safety requirements on developers of frontier AI models - those trained with significant computing power - to prevent catastrophic harms such as cyberattacks or the creation of biological weapons. The bill passed the California State Legislature in August 2024 but was vetoed by Governor Gavin Newsom on September 29, 2024, making California the first U.S. state to attempt comprehensive AI safety legislation, though the veto left the regulatory landscape unresolved.
The bill's core provisions targeted developers of AI models trained using more than 10^26 floating-point operations (FLOPs), a threshold that at the time encompassed only the most advanced systems from leading labs. It required such developers to implement a written safety and security framework, conduct pre-deployment testing, and ensure that models could be fully shut down in emergencies. The bill also established a new state agency, the Frontier Model Division, within the Department of Technology, to enforce compliance and issue penalties for violations. Notably, it included a whistleblower protection clause for employees who reported safety concerns, and it mandated that developers annually certify their compliance with the state attorney general.
Legislative Journey and Key Amendments
SB 1047 underwent significant revisions during its passage through the California Senate and Assembly. Initially introduced in February 2024, the bill was amended in April to narrow its scope, focusing on models with training costs exceeding $100 million. Subsequent amendments in August clarified that the bill would not apply to open-source models unless they were fine-tuned or modified to meet the threshold, and it exempted models used solely for research or national security purposes. The bill also included a "kill switch" requirement, mandating that developers implement a capability to fully halt model operation, and it required the creation of a "safety framework" that would be publicly available.
The bill passed the Senate on May 21, 2024, with a 32-1 vote, and the Assembly on August 28, 2024, with a 45-11 vote. The final version was approved by the Senate on August 30, 2024, and sent to Governor Newsom. Throughout the process, the bill attracted intense lobbying from both proponents and opponents, with tech companies and academic institutions weighing in on its potential impact on innovation.
Support and Opposition
Supporters of SB 1047, including prominent AI researchers such as Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, argued that the bill was a necessary precaution against the potential existential risks posed by advanced AI. They contended that voluntary industry commitments were insufficient and that government regulation was essential to ensure safety. The bill was also backed by labor unions, consumer advocacy groups, and some tech executives, who saw it as a way to hold AI developers accountable for harms.
Opponents, including major AI companies like OpenAI, Anthropic, and Google DeepMind, as well as tech trade groups and some academics, argued that the bill was overly broad and would stifle innovation. They claimed that the FLOP threshold was arbitrary and that the bill would impose burdensome compliance costs, particularly on smaller startups. Some critics also argued that the bill's focus on hypothetical catastrophic risks distracted from more immediate harms like bias and misinformation. Notably, Anthropic initially expressed concerns but later offered a more supportive stance, suggesting amendments that were partially incorporated.
Governor Newsom's Veto and Rationale
On September 29, 2024, Governor Gavin Newsom vetoed SB 1047, citing concerns about its "overly broad" scope and its potential to "choke off" innovation. In his veto message, Newsom acknowledged the importance of AI safety but argued that the bill's focus on model size and compute thresholds was not the most effective approach. He noted that the bill could impose "unworkable" requirements on developers and that it did not adequately address the full range of AI risks, such as algorithmic discrimination and job displacement. Newsom instead called for a more balanced regulatory framework, pointing to ongoing executive orders and collaborations with experts to develop evidence-based policies.
The veto was met with mixed reactions. Supporters expressed disappointment, with Senator Wiener vowing to continue the fight in future sessions. Opponents praised the decision, with tech industry groups claiming that the veto protected California's leadership in AI. The veto also sparked debate about the role of state versus federal regulation, as the U.S. Congress had not yet passed comprehensive AI legislation.
Impact and Aftermath
Following the veto, the issue of AI regulation remained a prominent topic in California and nationally. In October 2024, Governor Newsom signed several other AI-related bills, including measures addressing deepfakes and the use of AI in healthcare, but he declined to enact broader safety legislation. The veto also influenced discussions in other states, with some lawmakers considering similar bills but adjusting their approaches based on California's experience.
The debate over SB 1047 highlighted the challenges of regulating rapidly evolving technology. Proponents of the bill argued that waiting for federal action would leave the public unprotected, while opponents contended that premature regulation could hinder beneficial developments. The bill's legacy includes raising awareness of AI safety issues and prompting companies to adopt voluntary safety commitments, such as those made at the White House AI Safety Summit in 2023.
Broader Context in AI Regulation
SB 1047 was part of a broader global movement toward AI regulation. The European Union's AI Act, which was finalized in 2024, took a risk-based approach, categorizing AI applications by risk level and imposing obligations accordingly. In contrast, SB 1047 focused specifically on frontier models, a distinction that some experts argued was more targeted but also more arbitrary. The bill's failure underscored the difficulty of crafting legislation that balances innovation and safety, a challenge that continues to occupy policymakers worldwide.
Technical and Ethical Considerations
The bill's reliance on FLOP thresholds raised technical questions about how to measure AI capabilities. Critics noted that compute is not a perfect proxy for risk, as smaller models can sometimes be fine-tuned to achieve dangerous capabilities. The bill also touched on ethical issues, such as the responsibility of developers for downstream uses of their models, and the tension between transparency and national security. These considerations remain central to ongoing debates about AI governance.
Future Prospects
As of early 2025, no similar bill has been enacted in California, but Senator Wiener has indicated that he will reintroduce legislation in the next session, possibly with revised criteria. Meanwhile, the federal government has taken steps to address AI safety, including an executive order from President Biden in October 2023 and the establishment of the U.S. AI Safety Institute. The outcome of these efforts will likely shape the future of AI regulation in the United States, with California's experience serving as a cautionary tale or a blueprint, depending on one's perspective.