# California SB 1047 Overview

California SB 1047 is a 2024 state bill imposing safety requirements on large AI models, including risk assessments, incident reporting, and liability for severe harms, targeting developers of frontier AI systems.

California Senate Bill 1047, known as the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, is a proposed state law that would establish binding safety obligations for developers of large-scale artificial intelligence models. Introduced in February 2024 by Senator Scott Wiener, the bill aims to mitigate catastrophic risks posed by advanced AI systems, including potential cyberattacks, bioweapon creation, and mass casualties. It passed the California State Senate in May 2024 and the Assembly in August 2024, but was vetoed by Governor Gavin Newsom on September 29, 2024, making it one of the most debated AI regulation efforts in the United States.

The legislation targets "covered models" - AI systems trained with computing power exceeding 10^26 floating-point operations (FLOPs), a threshold designed to capture frontier models like those developed by [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). Developers must implement a safety framework that includes risk assessment, incident reporting, and a kill switch to halt model operation if necessary. The bill also imposes liability on developers for severe harms caused by their models, with penalties up to $10 million for false statements and up to $50 million for actual harm. Exemptions apply to models used solely for research, open-source models with modifications, and models below the compute threshold, though the bill includes a provision for the California Department of Technology to adjust the threshold as computing power grows.

## Background and Legislative Context

The bill emerged amid growing concerns about the rapid advancement of [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) and [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) systems, which have demonstrated capabilities in coding, reasoning, and content generation. Proponents argued that voluntary commitments from AI labs were insufficient, citing incidents like the misuse of AI for disinformation and the potential for future systems to evade human control. California, as the home of many leading AI companies, was seen as a natural venue for pioneering regulation, given its history of setting national standards in areas like vehicle emissions and data privacy.

Senator Wiener introduced SB 1047 on February 7, 2024, with co-sponsorship from the Future of Life Institute and support from prominent AI researchers including [yoshua-bengio](https://www.wikiprompt.org/wiki/yoshua-bengio) and [geoffrey-hinton](https://www.wikiprompt.org/wiki/geoffrey-hinton). The bill underwent multiple amendments during committee hearings, addressing criticisms about overbreadth and unintended consequences. Notably, the definition of "covered model" was refined to exclude models that are not deployed or made available to the public, and the liability clause was narrowed to apply only when the developer knew or should have known of the risk.

## Key Provisions and Requirements

Under SB 1047, developers of covered models must submit a "Safety and Security Framework" to the California Department of Technology before beginning training. This framework must detail the model's intended uses, potential risks, and mitigation measures. Specific requirements include:

- **Risk Assessment**: Developers must conduct a pre-training risk assessment, evaluating the model's potential to enable cyberattacks, chemical or biological weapons, or other catastrophic harms. The assessment must be updated after training and before public release.
- **Incident Reporting**: Any "critical harm incident" - defined as an event causing death, mass casualties, or significant property damage - must be reported to the state attorney general within 72 hours. The report must include technical details and the developer's response.
- **Kill Switch**: The framework must include a "prompt and full shutdown" capability, allowing the developer to immediately halt the model's operation if it poses an imminent threat.
- **Third-Party Audits**: Developers must hire independent auditors to verify compliance with the framework, with audit reports submitted to the state annually.
- **Whistleblower Protections**: Employees who report safety violations are protected from retaliation, encouraging internal transparency.

The bill also prohibits developers from retaliating against employees who raise safety concerns, and it requires that all covered models be developed in a manner that minimizes the risk of "model theft" - unauthorized access to the model's weights.

## Liability and Enforcement

SB 1047 establishes a novel liability regime for AI developers. If a covered model causes severe harm - defined as death, mass casualties, or property damage exceeding $500 million - the developer can be held liable even if they did not intentionally cause the harm. This "strict liability" approach is similar to product liability laws for defective goods, but it applies to software, which has traditionally been exempt from such standards.

Enforcement would be handled by the California Attorney General, who can seek civil penalties and injunctive relief. The bill also creates a "Frontier AI Division" within the Department of Technology, tasked with overseeing compliance and maintaining a public registry of covered models. Penalties for false statements in safety frameworks are up to $10 million per violation, while actual harm can result in fines up to $50 million. However, the bill explicitly states that it does not create a private right of action, meaning individuals cannot sue developers directly under this law.

## Support and Opposition

The bill drew polarized reactions. Supporters, including the Center for AI Safety and the Future of Life Institute, argued that it provides essential guardrails for a technology with existential risks. They pointed to surveys showing that a majority of AI researchers believe there is a significant chance of catastrophic outcomes from advanced AI. Supporters also noted that the bill's compute threshold ensures it only affects the largest labs, sparing startups and academic institutions.

Opponents came from two main camps. Industry groups, including the Chamber of Commerce and several AI companies, argued that the bill would stifle innovation and drive AI development out of California. They claimed that the strict liability standard would discourage open-source development, as even modified models could be traced back to the original developer. Some technologists, like [fei-fei-li](https://www.wikiprompt.org/wiki/fei-fei-li) and [andrew-ng](https://www.wikiprompt.org/wiki/andrew-ng), signed open letters opposing the bill, arguing that it was based on speculative fears rather than current capabilities.

A second group of critics, including some AI safety researchers, argued that the bill was too weak. They noted that the compute threshold could be circumvented by distributed training or by using more efficient algorithms, and that the liability provisions only apply to harms that are "reasonably foreseeable," which could be difficult to prove. Some also criticized the bill for not addressing smaller models that could still be dangerous in combination with other tools.

## Amendments and Compromises

During the legislative process, SB 1047 was amended to address some concerns. The original version required developers to certify that their models could not be used to create weapons of mass destruction, which critics called impossible to guarantee. This was changed to a "reasonable assurance" standard. The bill also added an exemption for models used exclusively for military or national security purposes, and it clarified that open-source models with fewer than 10^26 FLOPs of training compute are not covered.

Another significant amendment was the removal of a provision that would have created a new state agency to regulate AI. Instead, the bill relies on existing departments, with the Department of Technology taking the lead. This was seen as a concession to opponents who feared bureaucratic overreach.

## Governor's Veto and Aftermath

Governor Newsom vetoed SB 1047 on September 29, 2024, in a move that surprised many observers. In his veto message, Newsom argued that the bill was "well-intentioned" but overly broad, and that it would impose "rigid, costly, and potentially fatal" requirements on AI developers without adequate federal coordination. He called for a more "flexible" approach, suggesting that California should work with the federal government and other states to develop a unified regulatory framework.

The veto was celebrated by industry groups and some academics, but disappointed safety advocates. Senator Wiener vowed to reintroduce the bill in the next legislative session, possibly with revisions. In the meantime, California has continued to pursue other AI regulations, including a law requiring transparency in AI-generated content and a task force studying AI's impact on employment.

## Broader Implications

SB 1047's trajectory reflects the broader debate over AI regulation in the United States. While the European Union passed the AI Act in 2024, the U.S. has relied largely on voluntary commitments from companies. The bill's failure highlights the difficulty of regulating a rapidly evolving technology, where definitions can become outdated within months. However, its introduction has already influenced corporate behavior, with several major AI labs announcing voluntary safety frameworks that mirror SB 1047's requirements.

The bill also raised questions about the role of state governments in regulating technology. Supporters argued that California has a responsibility to protect its citizens, while opponents warned of a patchwork of state laws that would burden companies. The debate is likely to continue, especially as AI capabilities grow and the potential for harm becomes more concrete. As of early 2025, no similar bill has passed in any other state, but the issue remains a priority for lawmakers.

## Technical and Ethical Considerations

The bill's compute threshold of 10^26 FLOPs is notable because it aligns with estimates of the training cost for models like GPT-4, which is believed to have used around 2.1e25 FLOPs. This means that only a handful of models would be covered, including those developed by [openai](https://www.wikiprompt.org/wiki/openai), [anthropic](https://www.wikiprompt.org/wiki/anthropic), and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). However, the threshold is not static; the bill includes a provision for the Department of Technology to adjust it based on trends in computing power, ensuring that it remains relevant as hardware improves.

Ethically, the bill raises questions about responsibility for AI actions. Unlike traditional software, where developers are generally not liable for user misuse, SB 1047 would hold developers accountable for foreseeable harms. This shift reflects a growing consensus that AI systems are not neutral tools but have agency, and that their creators have a duty of care. Critics argue that this could lead to over-cautious development, slowing progress on beneficial applications like medical diagnosis or climate modeling.

The bill also touches on the issue of [model-pruning](https://www.wikiprompt.org/wiki/model-pruning) and [data-augmentation](https://www.wikiprompt.org/wiki/data-augmentation), as developers might try to reduce a model's compute footprint to avoid regulation. However, the bill's definition of "covered model" includes any model that is a derivative of a covered model, making it difficult to circumvent. This "derivative" clause was a key point of contention, with open-source advocates arguing that it would chill innovation in the [open-panel](https://www.wikiprompt.org/wiki/open-panel) community.

## Conclusion

California SB 1047 represents a landmark attempt to regulate frontier AI, combining elements of product safety, environmental law, and technology policy. Although it was vetoed, the bill has set a precedent for future legislative efforts, both in California and elsewhere. Its core ideas - compute thresholds, safety frameworks, and liability for catastrophic harms - are likely to reappear in some form, whether at the state or federal level. The debate over SB 1047 has also energized public discourse on AI safety, bringing issues like [rlaif](https://www.wikiprompt.org/wiki/rlaif) and [gradient-clipping](https://www.wikiprompt.org/wiki/gradient-clipping) into the mainstream conversation. As AI continues to advance, the question of how to govern it will remain one of the most pressing challenges of the 21st century.

## References

- California State Senate. (2024). SB 1047: Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. Legislative Counsel's Digest.
- Wiener, S. (2024). Press release on SB 1047 introduction.
- Newsom, G. (2024). Veto message for SB 1047.
- Future of Life Institute. (2024). Analysis of SB 1047.
- Center for AI Safety. (2024). Statement on SB 1047.

---
Source: https://www.wikiprompt.org/wiki/california-sb-1047-overview
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-12T16:23:22.86265+00:00