The Center for AI Safety (CAIS) is an American nonprofit organization based in San Francisco that promotes the safe development and deployment of artificial intelligence. CAIS' work encompasses research in technical AI safety and AI ethics, advocacy, and support to grow the AI safety research field. It was founded in 2022 by Dan Hendrycks and Oliver Zhang.
In May 2023, CAIS published the statement on AI risk of extinction signed by hundreds of professors of AI, leaders of major AI companies, and other public figures. The organization operates as a hub for researchers, policymakers, and industry partners focused on mitigating potential harms from advanced Artificial intelligence systems.
Research
CAIS researchers published "An Overview of Catastrophic AI Risks", which details risk scenarios and risk mitigation strategies. Risks described include the use of AI in autonomous warfare or for engineering pandemics, as well as AI capabilities for deception and hacking. Another work, conducted in collaboration with researchers at Carnegie Mellon University, described an automated way to discover adversarial attacks of large language models, that bypass safety measures, highlighting the inadequacy of current safety systems.
The organization's technical research spans areas such as Machine learning robustness, neural network interpretability, and alignment of deep learning systems. CAIS researchers have also examined failure modes in generative AI and proposed frameworks for evaluating dangerous capabilities before deployment.
Activities
In 2023, CAIS' statement on AI risk of extinction gained the support of Anthropic's Dario Amodei, OpenAI's Sam Altman, and other AI industry leaders. That same year, the cryptocurrency exchange FTX (which went bankrupt in November 2022) attempted to recoup $6.5 million that it had donated to CAIS earlier that year.
Other initiatives include a compute cluster to support AI safety research, an online course titled "Intro to ML Safety", and a fellowship for philosophy professors to address conceptual problems. The Center for AI Safety Action Fund is a sponsor of the California bill SB 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act.
Advocacy and Policy
CAIS engages with policymakers at state and federal levels to promote regulatory frameworks for frontier AI systems. The organization has published briefings on topics including model safety evaluations, reinforcement learning from human feedback, and the risks of transformer-based architectures. Its advocacy efforts emphasize the need for pre-deployment testing and transparency standards.
Community Building
The center supports the broader AI safety ecosystem through grants, workshops, and collaborations with academic institutions such as Berkeley AI Research and Oxford University. It also maintains a directory of safety researchers and organizes regular seminars that bring together experts from Google DeepMind, Anthropic, and other leading labs.
See Also
References
Source facts from Wikipedia (CC BY-SA).
External Links
Official website