The Machine Intelligence Research Institute (MIRI), formerly the Singularity Institute for Artificial Intelligence (SIAI), is a non-profit research institute focused since 2005 on identifying and managing potential existential risks from artificial intelligence. MIRI's work has focused on a friendly AI approach to system design and on predicting the rate of technology development.
Founded in 2000 by Eliezer Yudkowsky, MIRI initially aimed to accelerate the development of artificial intelligence, but shifted its focus to risk management after concerns about superintelligent systems. The institute is based in Berkeley, California, and has been influential in the field of AI safety, particularly within the effective altruism movement.
History
In 2000, Eliezer Yudkowsky founded the Singularity Institute for Artificial Intelligence with funding from Brian and Sabine Atkins, with the purpose of accelerating the development of AI. However, Yudkowsky began to be concerned that AI systems developed in the future could become superintelligent and pose risks to humanity. In 2005, the institute moved from Atlanta to Silicon Valley and began to focus on ways to identify and manage those risks, which were at the time largely ignored by scientists in the field.
Starting in 2006, the institute organized the Singularity Summit to discuss the future of AI including its risks, initially in cooperation with Stanford University and with funding from Peter Thiel. The San Francisco Chronicle described the first conference as a "Bay Area coming-out party for the tech-inspired philosophy called transhumanism". In 2011, its offices were four apartments in downtown Berkeley. In December 2012, the institute sold its name, web domain, and the Singularity Summit to Singularity University, and in the following month took the name "Machine Intelligence Research Institute".
In 2014 and 2015, public and scientific interest in the risks of AI grew, increasing donations to fund research at MIRI and similar organizations. In 2019, Open Philanthropy recommended a general-support grant of approximately $2.1 million over two years to MIRI. In April 2020, Open Philanthropy supplemented this with a $7.7M grant over two years. In 2021, Vitalik Buterin donated several million dollars' worth of Ethereum to MIRI.
Research and Approach
MIRI's approach to identifying and managing the risks of AI, led by Yudkowsky, primarily addresses how to design friendly AI, covering both the initial design of AI systems and the creation of mechanisms to ensure that evolving AI systems remain friendly. MIRI researchers advocate early safety work as a precautionary measure. However, MIRI researchers have expressed skepticism about the views of singularity advocates like Ray Kurzweil that superintelligence is "just around the corner".
MIRI has funded forecasting work through an initiative called AI Impacts, which studies historical instances of discontinuous technological change, and has developed new measures of the relative computational power of humans and computer hardware. MIRI aligns itself with the principles and objectives of the effective altruism movement.
Notable Publications
MIRI staff have contributed to several influential papers and books. Notable works include:
- "Why We Should Be Concerned About Artificial Superintelligence" by Matthew Graves (2017), published in Skeptic.
- "Program Equilibrium in the Prisoner's Dilemma via Löb's Theorem" by Patrick LaVictoire, Benja Fallenstein, Eliezer Yudkowsky, et al. (2014), presented at the AAAI-14 Workshop.
- "Cheating Death in Damascus" by Nate Soares and Benjamin A. Levinstein (2017), presented at the Formal Epistemology Workshop.
- "Corrigibility" by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky, and Stuart Armstrong (2015), presented at the AAAI Workshops.
- "Aligning Superintelligence with Human Interests: A Technical Research Agenda" by Nate Soares and Benja Fallenstein (2015), published in The Technological Singularity: Managing the Journey.
- "Artificial Intelligence as a Positive and Negative Factor in Global Risk" by Eliezer Yudkowsky (2008), published in Global Catastrophic Risks.
- "Quantilizers: A Safer Alternative to Maximizers for Limited Optimization" by Jessica Taylor (2016), presented at the Thirtieth AAAI Conference on Artificial Intelligence.
- "Complex Value Systems in Friendly AI" by Eliezer Yudkowsky (2011), published in Artificial General Intelligence: 4th International Conference.
- "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All" by Eliezer Yudkowsky and Nate Soares (2025), a book published by Hachette.
Impact and Reception
MIRI has been a pioneer in the field of AI safety, drawing attention to existential risks from AI before it became a mainstream concern. Its research has influenced other organizations and researchers, including those at OpenAI and Anthropic. However, MIRI has also faced criticism for its theoretical focus and for perceived alarmism. Despite this, its work has been supported by major donors, including Open Philanthropy and Vitalik Buterin, reflecting its influence in the effective altruism community.