You can buy physical copies of the map – including shirts, mugs, stickers etc. AISafety.com makes no profit on purchases.
Category
04 December 2025
Suggest something to be listed on the map
Let us know of changes to a listing
AI Safety Awareness Project (AISAP)
Raising awareness about modern AI, highlighting its benefits and risks, and letting the public know how they can help – mainly through workshops.
Advocacy
Collective Action for Existential Safety (CAES)
Aiming to catalyze collective effort towards reducing existential risk, including through an extensive action list for individuals, organizations, and nations.
Existential Risk Observatory (ERO)
Informing the public debate on existential risks, on the basis that awareness is the first step towards reducing those risks.
Explainable
Empowering organizations, researchers, and creators to translate the future of AI into stories the world understands.
Global AI Moratorium (GAIM)
Calling on policymakers to implement a global moratorium on large AI training runs until alignment is solved.
Legal Advocates for Safe Science and Technology (LASST)
Researching ways to use legal advocacy to make science and technology safer, informing legal professionals about how to help, and advocating in the courts and policy-setting institutions.
Legal Safety Lab
Using legal advocacy in Europe to address risks from frontier technologies, working to foster responsible development and implementation practices.
PauseAI
Campaign group aiming to convince governments to pause AI development – through public outreach, engaging with decision-makers, and organizing protests.
Stop AI
Non-violent civil resistance organization working to permanently ban the development of smarter-than-human AI.
The Alliance for Secure AI
Communications nonprofit focused on changing the narrative air that policymakers in Washington D.C. breathe with respect to AI safety.
The Midas Project (TMP)
Watchdog nonprofit monitoring tech companies, countering corporate propaganda, raising awareness about corner-cutting, and advocating for the responsible development of AI.
Future of Life Institute (FLI)
Steering transformative technology towards benefitting life and away from extreme large-scale risks through outreach, policy advocacy, grantmaking, and event organization.
Advocacy, Governance, Funding
Buddhism & AI Initiative
A collaborative effort to bring together Buddhist communities, technologists, and contemplative researchers worldwide to help shape the future of AI.
Advocacy, Governance, Strategy
AI Safety Foundation
Conducting educational initiatives, research support, and public awareness campaigns. Project of Geoffrey Hinton.
Advocacy, Research support
AI Frontiers
Platform from the Center for AI Safety (CAIS) posting articles written by experts from a wide range of fields discussing the impacts of AI.
Blog
AI Policy Bulletin
Publishes policy-relevant perspectives on frontier AI governance, including research summaries, opinions, interviews, and explainers.
AI Prospects
Blog by Eric Drexler on AI prospects and their surprising implications for technology, economics, environmental concerns, and military affairs.
AI Safety Takes
Blog by Daniel Paleka, an AI safety researcher, discussing AI safety news. Posts about every two months.
Astral Codex Ten (ACX)
Blog covering many topics, including reasoning, science, psychiatry, medicine, ethics, genetics, economics, politics, and AI. Often book summaries and commentary on AI safety.
Bounded Regret
Blog on AI safety by Jacob Steinhardt, a UC Berkeley statistics professor, analysing risks, forecasting future breakthroughs, and discussing alignment strategies.
ChinaTalk
Deep, thoughtful coverage on China, technology, and US-China relations – written by Jordan Schneider.
Cold Takes
Blog about transformative AI, futurism, research, ethics, philanthropy etc. by Holden Karnofsky. Includes the "Most Important Century" post series. Last posted in 2024.
DeepMind Safety Research
Blog from the Google DeepMind safety team discussing research ideas about building AI safely.
Don't Worry about the Vase
Blog by Zvi Mowshowitz on various topics, including AI, with detailed analysis, personal insights, and a rationalist perspective. Posts very frequently.
Miles's Substack
Blog from ex-OpenAI (now independent) AI policy researcher Miles Brundage on the rapid evolution of AI and the urgent need for thoughtful governance.
Obsolete
Reporting and analysis on capitalism, great power competition, and the race to build superintelligence by freelance journalist Garrison Lovely.
Paul Christiano's Blog
Blog on aligning prosaic AI by one of the leading AI safety researchers. No longer active but the archive is high quality.
The Field Building Blog (FBB)
Blog about building the fields of AI safety and Effective Altruism, discussing big-picture strategy and sharing personal experience from working in the space.
The Power Law
Top forecaster Peter Wildeford forecasts the future and discusses AI, national security, innovation, emerging technology, and the powers – real and metaphorical – that shape the world.
Threading the Needle
Publication by Anton Leicht on the political economy of AI progress, featuring weekly posts on how institutions and political incentives interface with fast technological change.
Victoria Krakovna's Blog
Blog by an AI safety researcher at Google DeepMind, covering alignment research, rationality, and personal productivity insights. Last AI post was 2023.
AISafety.com: Media channels
The AI safety space is changing rapidly. This directory of key information sources can help you keep up to date with the latest developments.
Blog, Newsletter, Podcast, Video
DeepSeek
Chinese capabilities lab developing and releasing open-weights large language models. Created DeepSeek-R1.
Capabilities research
xAI
Capabilities lab led by Elon Musk with the mission of advancing our collective understanding of the universe. Created Grok.
Obelisk
Team of researchers housed at Astera pursuing an exploratory, neuroscience-informed approach to engineering AGI.
Capabilities research, Conceptual research
Anthropic
Capabilities company focusing on LLM alignment, particularly interpretability. Featuring Chris Olah, Jack Clark, and Dario Amodei. Created Claude.
Capabilities research, Empirical research
Cyborgism
A strategy for accelerating alignment research by using human-in-the-loop systems which empower human agency rather than outsource it.
Google DeepMind
London-based capabilities company with a strong safety team, led by Demis Hassabis. Created AlphaGo, AlphaFold, and Gemini.
Gray Swan
For-profit company developing tools that automatically assess the risks of AI models and developing its own AI models aiming to provide best-in-class safety and security.
OpenAI
Capabilities company that created ChatGPT, led by Sam Altman. Throughout 2024, roughly half of then-employed AI safety researchers left the company.
Safe Superintelligence Inc. (SSI)
Research lab founded by Ilya Sutskever, comprised of a small team of engineers and researchers working towards building a safe superintelligence.
80,000 Hours Career Guide
Regularly-updated article with motivation and advice around pursuing a career in AI safety.
Career support
80,000 Hours Job Board
Curated list of job posts around the world tackling pressing problems, including AI safety. Also has a newsletter.
AI Safety Quest
Grassroots volunteer organization helping people contribute to reducing catastrophic risk from AI by directing them to the most relevant resources and communities.
AISafety.com: Advisors
Directory of advisors offering free guidance calls to help you discover how best to contribute to AI safety, tailored to your skills and interests.
Effective Thesis
Empowering students to use their theses as a pathway to impact. Lists research topic ideas and runs an accelerator and fellowship coaching people working on them.
High Impact Professionals (HIP)
Supporting working professionals to maximize their positive impact through their talent directory and Impact Accelerator Program.
How to pursue a career in technical AI alignment
A guide written for people who are familiar with the arguments for the importance of AI alignment and are considering pursuing a career working on it.
Probably Good
Helps those who want to have a meaningful impact with their careers brainstorm career paths, evaluate options, and plan next steps.
Successif
Helping professionals transition to high-impact work by performing market research on impactful jobs and providing career mentoring, opportunity matching, and professional training.
Upgradable
Nonprofit helping existential safety advocates to systematically optimize their lives and work.
Arcadia Impact
Runs various projects aimed at education, skill development, and creating pathways into impactful careers.
Career support, Training and education
Alignment Research Center (ARC)
Research organization trying to understand how to formalize mechanistic explanations of neural network behavior.
Conceptual research
Alignment of Complex Systems Research Group (ACS)
Studying questions about multi-agent systems composed of humans and advanced AI. Based at Charles University, Prague.
Arbital
Wiki on AI alignment theory, mostly written by Eliezer Yudkowsky. Includes foundational concepts, open problems, and proposed solutions.
Center for Human-Compatible AI (CHAI)
Developing the conceptual and technical wherewithal to reorient the general thrust of AI research towards provably beneficial systems. Led by Stuart Russell at UC Berkeley.
Computational Rational Agents Laboratory (CORAL)
Research group studying agent foundations in order to create the mathematical tools to align the objectives of an AI with human values.
Dovetail
Research group working on foundational mathematics that provides an understanding of the nature of AI agents.
Dr. Roman Yampolskiy
Professor at University of Louisville with a background in cybersecurity, and author of over 100 publications – including 2 books on AI safety.
Dylan Hadfield-Menell
Assistant professor at MIT working on agent alignment. Runs the Algorithmic Alignment Group.
John Wentworth
Independent alignment researcher working on selection theorems, abstraction, and agency.
Modeling Cooperation
Conducting long-term future research on improving cooperation in competition for the development of transformative AI.
Orthogonal
Formal alignment organization led by Tamsin Leake, focused on agent foundations. Also has a public Discord server.
Steve Byrnes's Brain-Like AGI Safety
Brain-inspired framework using insights from neuroscience and model-based reinforcement learning to guide the design of aligned AGI systems.
Team Shard
Small group of independent researchers trying to find reward functions which reliably instill certain values in agents.
Softmax
Research organization developing a theory of "organic alignment" to foster adaptive, non-hierarchical cooperation between humans and digital agents.
Conceptual research, Empirical research
AE Studio
Large team taking a 'Neglected Approaches' approach to alignment, tackling the problem from multiple, often overlooked angles in both technical and policy domains.
Conceptual research, Empirical research, Governance
Formation Research
Aiming to reduce lock-in risks by researching fundamental lock-in dynamics and implementing high-impact interventions.
Frontier AI Research (FAIR)
Argentine nonprofit conducting both theoretical and empirical research to advance frontier AI safety as a sociotechnical challenge.
MIT Algorithmic Alignment Group
Working towards better conceptual understanding, algorithmic techniques, and policies to make AI safer and more socially beneficial.
AI Alignment Forum
Hub for researchers to discuss all ideas related to AI safety. Discussion ranges from technical models of agency to the strategic landscape, and everything in between.
Conceptual research, Empirical research, Governance, Strategy
Association for Long Term Existence and Resilience (ALTER)
Israeli research and advocacy nonprofit working to investigate, demonstrate, and foster useful ways to safeguard and improve the future of humanity.
Conceptual research, Governance
Equilibria Network
Using category theory and agent-based simulations to predict where multi-agent AI systems break down and identify control levers to prevent collective AI behavior from being harmful.
Center on Long-Term Risk (CLR)
Research, grants, and community-building around AI safety, focused on conflict scenarios as well as technical and philosophical aspects of cooperation.
Conceptual research, Strategy, Funding
Iliad
Applied mathematics research nonprofit dedicated to advancing foundational alignment research. Organizes events and a research residency.
Conceptual research, Training and education
Aether
Aiming to conduct technical research that yields valuable insights into the risks and opportunities that LLM agents present for AI safety.
Empirical research
Aligned AI
Oxford-based startup attempting to use mathematical and theoretical techniques to achieve safe off-distribution generalization.
Apollo Research
Aiming to detect deception by designing AI model evaluations and conducting interpretability research to better understand frontier models. Also provides guidance to policymakers.
Cadenza Labs
Researching, benchmarking, and developing lie detectors for LLMs, including using cluster normalization techniques.
Cavendish Labs
AI safety (and pandemic prevention) research community based in a small town in Vermont, USA.
Compassion in Machine Learning (CaML)
Working on research to make transformative AI more compassionate towards all sentient beings through new data generation methods.
Computational and Biological Learning Lab (CBL)
Research group using engineering approaches to understand the brain and to develop artificial learning systems.
Conjecture
Alignment startup born out of EleutherAI, following a Cognitive Emulation approach to build controllable LLMs and tackle core AI safety challenges.
Coordinal Research
Developing tools that accelerate the rate human researchers can make progress on alignment, and building automated research systems that can assist in alignment work today.
EquiStamp
Engineering company providing evaluation implementation, data annotation, and project operations so AI safety researchers can focus on research.
FAR.AI
Conducting research, hosting events, and running programs – including the FAR.Lab coworking space.
Geodesic
Research nonprofit focused on leading projects with the shortest path to impact for AI safety – currently working on chain-of-thought health and monitoring.
Goodfire
Mechanistic interpretability lab building infrastructure to allow researchers to decode neural networks in order to make them more understandable, editable, and safer.
Krueger AI Safety Lab (KASL)
AI safety research group at Mila, led by David Krueger. Previously based at the University of Cambridge.
LawZero
Advancing research and developing technical solutions for safe-by-design AI systems based on Scientist AI, a research direction led by Yoshua Bengio.
Luthien
Developing AI Control into production-ready solutions that can be deployed in real-world AI systems.
Meaning Alignment Institute (MAI)
Researching how to align AI, markets, and democracies with what people value – from theory to practice.
Model Evaluation & Threat Research (METR)
Researching, developing, and running evaluations of AI capabilities, including broad autonomous capabilities and the ability of AI systems to conduct AI R&D.
NYU Alignment Research Group (ARG)
Group of researchers at New York University doing empirical work with LLMs in order to address risks posed by advanced AI.
Ought
Product-driven research lab developing mechanisms for delegating high-quality reasoning to ML systems. Built Elicit, an AI assistant for researchers and academics.
Palisade Research
Investigating cyber offensive AI capabilities and the controllability of frontier AI models in order to advise policy makers and the public on AI risks.
Redwood Research
Nonprofit researching AI control and alignment faking. Also consults governments and AI companies on AI safety practices.
Simplex
Small team of researchers and engineers aiming to bring the best of physics and computational neuroscience together in order to understand and control AGI.
Transluce
Building open source, scalable, AI-driven tools to understand and analyze AI systems and steer them in the public interest.
TruthfulAI
Nonprofit led by Owain Evans, researching situational awareness, deception, and hidden reasoning in LLMs.
AI Objectives Institute (AOI)
Creating tools and programs to ensure that AI and economic systems of the future are built with genuine human objectives.
Empirical research, Capabilities research
EleutherAI
Open-source research lab focused on interpretability and alignment. Operates primarily through a public Discord server, where research is discussed and projects are coordinated.
Workshop Labs
For-profit working on mitigating gradual disempowerment and the intelligence curse by creating personalized models.
Timaeus
Using singular learning theory to develop the science of understanding how training data shapes AI model behavior.
Empirical research, Conceptual research
Center for AI Safety (CAIS)
Conducting safety research, building the field of AI safety researchers, and advocating for safety standards.
Empirical research, Conceptual research, Advocacy
Center for Long-Term Cybersecurity (CLTC)
UC Berkeley research center bridging academic research and practical policy needs in order to anticipate and address emerging cybersecurity challenges.
Empirical research, Governance, Strategy
Harmony Intelligence
For-profit creating defensive cybersecurity products that mitigate AI x cyber risk. Also supports the Australian government with AI safety evals and policymaking.
Empirical research, Strategy
AI Futures Project (AIFP)
Small research group forecasting the future of AI. Created 'AI 2027', a detailed forecast scenario projecting the development of artificial superintelligence.
Forecasting
Forecasting Research Institute (FRI)
Advancing the science of forecasting for the public good by working with policymakers and nonprofits to design practical forecasting tools, and test them in large experiments.
MIT FutureTech
Interdisciplinary group aiming to identify and understand trends in computing that create opportunities for (or pose risks to) our ability to sustain economic growth.
Manifold Markets
Prediction market platform covering many topics, using play money called "mana". Features markets on AI and AI safety.
Metaculus
Well-calibrated forecasting platform covering a wide range of topics, including AI and AI safety.
Quantified Uncertainty Research Institute (QURI)
Advancing forecasting and epistemics to improve the long-term future of humanity. Conducts research and builds software.
Transformative Futures Institute (TFI)
Research nonprofit utilizing foresight to mitigate societal-scale risks from advanced AI and other potential global catastrophic risks.
Epoch AI
Research institute examining the driving forces behind AI and forecasting its economic and societal impact.
Forecasting, Strategy
AE Studio Research
Empowering innovators and scientists to increase human agency by creating the next generation of responsible AI. Providing support, resources, and open-source software.
Funding
AI Risk Mitigation (ARM) Fund
Aiming to reduce catastrophic risks from advanced AI through grants towards technical research, policy, and training programs for new researchers.
AI Safety Advocacy Grants
Grant program run by Nonlinear, providing funding to those raising awareness about AI risks or advocating for a pause in AI development.
AI Safety Science
Funder housing the science-focused philanthropic efforts of Eric and Wendy Schmidt, moving large amounts of funding towards AI safety via its AI institute.
AI Safety Tactical Opportunities Fund (AISTOF)
Pooled multi-donor fund structured to be fast and rapidly capture emerging opportunities, including in governance, technical alignment, and evaluations. Managed by JueYan Zhang.
AI2050
Philanthropic initiative supporting researchers working on key opportunities and hard problems that are critical to get right for society to benefit from AI.
AISafety.com: Donation Guide
Regularly-updated guide on how to donate most effectively to the AI safety field given the funding and time you have available.
AISafety.com: Funding
Comprehensive and up-to-date directory of sources of financial support for AI safety projects, ranging from grant programs to venture capitalists.
Advanced Research + Invention Agency (ARIA)
UK government R&D funding agency aiming to unlock scientific and technological breakthroughs that benefit everyone. Similar to DARPA in the US.
An Overview of the AI Safety Funding Situation
An analysis of the main funding sources in AI safety over time, useful for gaining a better understanding of what opportunities exist in the space.
Astralis Foundation
Funding initiative with $25M annual giving, backing people and ideas with the funding, strategic guidance, and networks they need to steer transformative AI toward beneficial outcomes.
Center on Long-Term Risk (CLR): Fund
Supports projects and individuals aiming to address worst-case suffering risks from the development and deployment of advanced AI systems.
Coefficient Giving
The largest funder in the existential risk space, backed primarily by Dustin Moskovitz and Cari Tuna. Previously called Open Philanthropy.
Cooperative AI Foundation (CAIF)
Charity foundation backed by a large philanthropic commitment supporting research into improving cooperative intelligence of advanced AI.
EA Infrastructure Fund (EAIF)
Aiming to increase the impact of effective altruism projects (including AI safety) by increasing their access to talent, capital, and knowledge.
Ergo Impact
Helping philanthropists find, fund, and scale the most promising people and solutions to the world’s most pressing problems.
Foresight Institute: Funding
Foresight's 'AI for Science & Safety Nodes' program offers funding, a community hub, and local compute in either San Francisco or Berlin.
Future of Life Foundation (FLF)
Accelerator aiming to steer transformative technology towards benefiting life and away from extreme large-scale risks.
Future of Life Institute (FLI): Fellowships
Includes PhD and postdoctoral fellowships in technical AI safety, and a PhD fellowship in US-China AI governance.
Giving What We Can (GWWC)
Community of donors who have pledged to donate a significant portion of their income to highly effective charities, including those in AI safety.
Lionheart Ventures
VC firm investing in ethical founders developing transformative technologies that have the potential to impact humanity on a meaningful scale.
Long-Term Future Fund (LTFF)
Making grants addressing global catastrophic risks, promoting longtermism, and otherwise increasing the likelihood that future generations will flourish.
Longview Philanthropy
Devises and executes bespoke giving strategies for major donors, working with them at all stages of their giving journey.
Macroscopic Ventures
Swiss VC focused on reducing suffering risks, including that posed by catastrophic AI misuse and AI conflict.
Manifund
Marketplace for funding new charities, including in AI safety. Users can find impactful projects, buy impact certificates, and weigh in on what gets funded.
Meta Charity Funders
Network of donors funding charitable projects that work one level removed from direct impact, often cross-cutting between cause areas.
Mythos Ventures
VC aiming to empower founders building a radically better world with safe AI systems by investing in ambitious teams with defensible strategies that can scale to post-AGI.
Saving Humanity from Homo Sapiens (SHfHS)
Small organization with a long history of finding the people doing the best work to prevent human-created existential risks and financially supporting them.
Survival and Flourishing Fund (SFF)
The second largest funder in AI safety, using an algorithm and meeting procedure called 'The S-process' to allocate grants.
The Navigation Fund
Jed McCaleb's fund, making grants to organizations and projects in various cause areas – including AI safety.
Halcyon Futures
Identifying leaders from business, policy, and academia, and helping them take on ambitious projects in AI safety.
Funding, Career support
Juniper Ventures
VC firm investing in AI safety startups. Run by exited founders and backed by Reid Hoffman, Eric Ries, and Geoff Ralston.
Oxford Martin AI Governance Initiative (AIGI)
Research center housed in the Martin School of the University of Oxford researching AI governance from both technical and policy perspectives.
Governance
AI Governance & Safety Canada (AIGS Canada)
Nonpartisan nonprofit and community of people across Canada, producing white papers, legislative recommendations, and government submissions.
AI Policy Institute (AIPI)
Channeling public concern around AI into effective regulation through engaging with policymakers, media, and the public.
AI Standards Lab
Politically and geographically neutral nonprofit converting insights from the existing literature into ready-made text for AI safety standards.
Americans for Responsible Innovation (ARI)
Bipartisan nonprofit seeking to address a broad range of policy issues raised by AI, including current harms, national security concerns, and emerging risks.
Beijing Institute of AI Safety and Governance (Beijing-AISI)
Developing AI safety and governance frameworks in order to provide a safe foundation for AI innovation and applications.
Center for AI Standards and Innovation (CAISI)
US government organization developing voluntary AI standards and conducting security evaluations of AI systems. Formerly the USAISI.
Center for Security and Emerging Technology (CSET)
Georgetown University think tank providing decision-makers with data-driven analysis on the security implications of emerging technologies.
Centre for Future Generations (CFG)
Brussels think tank focused on helping governments anticipate and responsibly govern the societal impacts of rapid technological change.
Centre for the Governance of AI (GovAI)
AI governance research group at Oxford, producing research tailored towards decision-makers and running career development programmes.
China Al Safety & Development Association (CnAISDA)
China’s self-described counterpart to the AI safety institutes of other countries. Its primary function is to represent China in international AI conversations.
European AI Office
Established within the European Commission as the centre of AI expertise in the EU, playing a key role in implementing the AI Act.
General Purpose AI Policy Lab
Producing research that helps French institutional actors address the security and international coordination requirements posed by the development of general purpose AI.
Good Ancestors
Australian lobbying organization focused on AI safety policy and other AI-related issues, including cybersecurity and biosecurity. Also runs Australians for AI Safety.
Institute for Law & AI (LawAI)
Think tank researching and advising on the legal challenges posed by AI, premised on the idea that sound legal analysis will promote security, welfare, and the rule of law.
RAND Global and Emerging Risks
Delivering rigorous and objective public policy research on the most consequential challenges to civilization and global security.
SaferAI
French nonprofit developing quantitative risk models, evaluating company practices, and leading standards development that shapes AI regulation worldwide.
Secure AI Project
Nonprofit developing and advocating for AI safety principles to be put into practice in US state and federal legislatures.
Simon Institute for Longterm Governance
Conducting research on international AI governance, facilitating exchange between technical and policy communities, and educating diplomats and civil servants about frontier AI.
The AI Policy Network (AIPN)
Building bipartisan support for federal policies that prepare the US for the emergence of AI systems on the path to AGI and beyond.
The AI Whistleblower Initiative (AIWI)
Helps AI insiders raise concerns about potential risks and misbehavior in AI development by providing whistleblowing services, expert guidance, and secure communication tools.
The Future Society (TFS)
Drawing on their analyses and networks to provide decision-makers worldwide with pragmatic guidance on navigating technological, catastrophic, and political risks from AI.
UK AI Security Institute (UK AISI)
UK government organization conducting research and building infrastructure to test the safety of advanced AI and measure its impacts. Also working to shape global policy.
Vista Institute for AI Policy
Promoting informed policymaking to navigate emerging challenges from AI through research, knowledge-sharing, and skill building.
ControlAI
Nonprofit fighting to keep humanity in control of AI by developing policy and conducting public outreach.
Governance, Advocacy
International AI Governance Alliance (IAIGA)
Nonprofit dedicated to establishing an independent global organization capable of effectively mitigating extinction risks from AI and fairly distributing its economic benefits to all.
Machine Intelligence Research Institute (MIRI)
The original AI safety technical research organization, co-founded by Eliezer Yudkowsky. Now focusing on policy and public outreach.
Governance, Advocacy, Conceptual research
Effective Institutions Project (EIP)
Research and advisory organization focused on improving the way institutions make decisions on critical global challenges.
Governance, Strategy
Institute for AI Policy and Strategy (IAPS)
Research and field-building organization focusing on policy and standards, compute governance, and international governance and China.
International Association for Safe & Ethical AI (IASEAI)
Connecting experts from academia, policy groups, civil society, industry, and beyond to promote research, shape policy, and build understanding around AI safety.
Centre pour la Sécurité de l'IA (CeSIA)
French nonprofit dedicated to research, education, and awareness-raising about the most extreme risks of advanced AI.
Governance, Strategy, Advocacy
AI Safety Events & Training
Weekly newsletter listing newly-announced AI safety events and training programs, both online and in-person.
Newsletter
AI Safety Newsletter
Newsletter published every few weeks discussing recent developments in AI and AI safety. No technical background required.
AI Safety at the Frontier
Johannes Gasteiger, an alignment researcher at Anthropic, selects and summarizes the most interesting AI safety papers each month.
AI Safety in China
Newsletter from Concordia AI, a Beijing-based social enterprise, providing updates on AI safety developments in China.
Can We Secure AI With Formal Methods?
Current-events newsletter for keeping up to date with FMxAI (formal methods and AI) with a gear toward safety, doing a mix of shallow technical reviews of papers and movement updates.
Import AI
Weekly updates on the latest developments in AI research (including governance) written by Jack Clark, co-founder of Anthropic.
ML Safety Newsletter
Infrequent newsletter aiming to inform readers about recent ML safety research, focussing on areas like adversarial robustness, interpretability, and control.
Transformer
Aiming to help decision-makers understand what’s happening in AI and why it matters – through news roundups, explainers, features, and opinion pieces.
AI Alignment Awards
Ran research paper/essay-writing contests to advance alignment. Last round was 2023.
No longer active
AI Safety Communications Centre (AISCC)
Connected journalists to AI safety experts and resources.
AI Safety Hub (ASH)
Supported individuals pursuing a career in AI safety by running AI Safety Hub Labs, where participants complete their first research project. Now run by LASR Labs.
AI Safety Ideas
Crowdsourced repository of possible research projects and testable hypotheses, run by Apart Research.
AI Safety Support (AISS) Newsletter
Listed opportunities in AI alignment. Dropped due to lack of a maintainer.
AI Timeline
Visual overview of the major events in AI over the last decade, from cultural trends to technical advancements. No longer maintained.
Arkose
AI safety field-building nonprofit. Ran support programs facilitating technical research, did outreach, and curated educational resources. Closed due to lack of funding.
Campaign for AI Safety (CAS)
Increased public understanding of AI safety and called for strong laws to stop the development of dangerous and overly powerful AI. Now merged with the Existential Risk Observatory.
Center for AI Policy (CAIP)
Nonpartisan research organization developing policy and conducting advocacy to mitigate catastrophic risks from AI. Shut down due to lack of funding.
Donations List Website
Minimalistic site focused on understanding donations, donors, donees, and the thoughts and discussions that go alongside donation decisions.
From AI to ZI
Blog on AI safety work by a PhD mathematician and AI safety researcher, Robert Huben. Currently dormant.
Future of Humanity Institute (FHI)
Longtermist/x-risk research organization led by Nick Bostrom at the University of Oxford. Killed by university politics disallowing them from fundraising.
GiveWiki
Crowdsourced charity evaluation and philanthropic networking platform that let funders, donors, and scouts discover and support high-impact projects. No longer active.
Global Priorities Institute (GPI)
University of Oxford research center conducting foundational research to inform the decision-making of those seeking to do as much good as possible. Closed in 2025.
Lightspeed Grants
Fast funding for projects that have a chance of substantially changing humanity's future trajectory for the better. Last funding round was 2023.
ML & AI Safety Updates
Weekly podcast, YouTube and newsletter with updates on AI safety. Dropped due to organizational reprioritization.
Machine Learning for Alignment Bootcamp (MLAB)
Bootcamp aimed at teaching ML relevant to doing alignment research. Run by Redwood Research for two iterations in 2022.
Nonlinear Network
Platform taking grant applications and presenting them to a donor circle, who then reached out if they were interested. Last round was in 2024.
OpenBook
Database of grants in effective altruism. No longer maintained.
Preamble Windfall Foundation
Funding organization aiming to minimize the risk of AI systems. Died for reasons unknown to us.
Safe AI London (SAIL)
Supported individuals in London interested in working on AI safety by raising awareness of the risks and providing high-quality resources and support.
Stop AGI
Website communicating the risks of god-like AI to the public and offering proposals on preventing its development.
Superlinear Prizes
Decentralized bounty platform for existential risk reduction and other effective altruist cause areas.
generative.ink
The blog of janus the GPT cyborg. Last post in 2023.
80,000 Hours Podcast
In-depth conversations about the world’s most pressing problems – including AI safety – and what you can do to help solve them.
Podcast
AI X-risk Research Podcast (AXRP)
Interviews with (mostly technical) AI safety researchers about their research, aiming to get a sense of why it was written and how it might reduce existential risk from AI.
Dwarkesh Podcast
Well-researched interviews with influential intellectuals going in-depth on AI and technology, and their broader societal implications.
Future of Life Institute (FLI) Podcast
Interviews with existential risk researchers, policy experts, philosophers, and a range of other influential thinkers.
The AI Policy Podcast
Podcast from the Center for Strategic & International Studies (CSIS) discussing AI regulation, innovation, national security, and geopolitics.
The Cognitive Revolution
Biweekly podcast where host Nathan Labenz interviews AI innovators and thinkers, discussing the transformative impact AI will likely have in the near future.
5050
13-week program run by Fifty Years, helping scientists and engineers build startups working on AI safety.
Research support
Alignment Ecosystem Development (AED)
Building and maintaining key online resources for the AI safety community, including AISafety.com and AISafety.info. Volunteers welcome.
Ashgro
Providing fiscal sponsorship to AI safety projects, saving them time and allowing them to access more funding.
Atlas Computing
Creating new AI safety orgs by identifying unowned problems, mapping stakeholders, drafting milestones, sourcing early funders, and recruiting an expert leader to take ownership.
Berkeley Existential Risk Initiative (BERI)
Providing flexible funding and operations support to university research groups working on existential risk, enabling projects otherwise hindered by university administration.
Catalyze Impact
Incubating early-stage AI safety research organizations. The program involves co-founder matching, mentorship, and seed funding, culminating in an in-person building phase.
Centre for Enabling EA Learning & Research (CEEALAR aka EA Hotel)
Free or subsidised accommodation and board in Blackpool, England, for people working on/transitioning to working on global catastrophic risks.
Constellation
Center for collaborative research in AI safety, supporting promising work through fellowships, workshops, and hosting individuals and teams.
European Network for AI Safety (ENAIS)
Community of researchers and policymakers from over 13 countries across Europe, united in their efforts to advance AI safety.
Flourishing Future Foundation (FFF)
Provides engineering teams, compute, and infrastructure to researchers advancing neglected approaches to alignment.
Future Matters
Conducts research and provides strategy consulting services to clients trying to advance AI safety – and other causes – through policy, politics, coalitions or social movements.
Good Impressions
Marketing firm helping socially impactful projects grow, both through consulting and running full-scale campaigns.
Impact Ops
Providing consultancy and hands-on support to help high-impact organizations upgrade their operations.
Lightcone Infrastructure
Nonprofit maintaining LessWrong, the Alignment Forum, and Lighthaven (an event space in Berkeley, USA).
London Initiative for Safe AI (LISA)
Coworking space hosting organizations (including BlueDot Impact, Apollo Research), acceleration programs (including MATS, ARENA), and independent researchers.
Mox
Incubator and coworking space in San Francisco aimed at various groups, including those working on AI safety.
Nonlinear
Nonprofit aiming to prevent extinction and reduce suffering risks by providing funding, 1-1 coaching, and career advice.
PEAKS
Coworking space in Zurich, Switzerland, for people working on AI safety or effective altruism. Regularly hosts events – ranging from relaxed gatherings to lightning talks and discussions.
Pause House
Free accomodation and board in Blackpool, England, for individuals doing work related to pushing for a pause to AI development.
RAISEimpact
Program designed to help AI safety organizations strengthen their management and leadership practices in order to amplify their effectiveness.
Rethink Wellbeing
Nonprofit running programs to give altruists the tools they need to improve their mental health, so they can have a greater impact on the world.
Safe AI Forum (SAIF)
Fostering responsible governance of AI to reduce catastrophic risks through shared understanding and collaboration among key global actors.
Singapore AI Safety Hub (SASH)
Coworking, events, and community space for people working on AI safety. Runs regular talks, hackathons, and networking events.
Trajectory Labs
AI safety coworking and events space in Toronto, Canada, providing a physical space, hosting events, and maintaining a collaborative community.
Apart Research
AI safety research lab hosting open-to-all research sprints, publishing papers, and incubating talented researchers.
Research support, Career support
Meridian
Coworking space and field-building nonprofit based in Cambridge, UK. Houses projects like CAISH and the ERA:AI Fellowship, as well as visiting AI safety researchers.
Research support, Training and education
AI Alignment Slack
The biggest real-time online community of people interested in AI safety, with channels ranging from general topics to specific fields to local groups.
Resource
AI Digest
Concise visual explainers of important trends in AI, grounded in concrete examples of what AI models can do right now.
AI Risk Explorer (AIRE)
Online platform monitoring the emergence of large-scale AI risks, featuring curated information on evaluations, incidents, and policies.
AI Risk: Why Care?
AI safety chatbot that excels at addressing hard questions and counterarguments about existential risk.
AI Safety Map Anki Deck
Flashcards for helping to learn and memorize the main organizations, projects, and programs currently operating in the AI safety space.
AI Safety Support (AISS)
Field-building organization now chiefly serving as the home for an extensive resources list called Lots of Links.
AI Watch
Database tracking people, organizations, and “products” in the AI safety community, serving as a reference for positions, affiliations, and related data.
AISafety.com
Hub for key resources for the AI safety community, including directories of courses, jobs, upcoming events and training programs etc. – and this map!
Effective Altruism Domains
Directory of domains freely available to be used for high-impact projects, including those contributing to AI safety.
Effective Altruism Forum
Forum on doing good as effectively as possible, including AI safety. Also has a podcast featuring text-to-speech narrations of top posts.
LessWrong
Online forum dedicated to improving human reasoning, containing a lot of AI safety content. Also has a podcast featuring text-to-speech narrations of top posts.
MIT AI Risk Repository
Comprehensive living database of over 1600 AI risks, categorized by their cause and risk domain.
AI Lab Watch
Collects actions frontier AI companies can take to improve safety and public information on what they're doing, then evaluates them accordingly.
Strategy
AI-Plans
Ranked and scored contributable compendium of alignment plans and their problems. Runs regular hackathons.
Center for AI Risk Management & Alignment (CARMA)
Conducting interdisciplinary research supporting global AI risk management. Also produces policy and technical research.
Centre for the Study of Existential Risk (CSER)
Interdisciplinary research center at the University of Cambridge dedicated to the study and mitigation of existential and global catastrophic risks.
Forethought
Small research nonprofit focused on how to navigate the transition to a world with superintelligent AI systems.
Global Catastrophic Risk Institute (GCRI)
Small think tank developing solutions for reducing existential risk by leveraging both scholarship and the demands of real-world decision-making.
Global Partnership on AI (GPAI)
International initiative with 44 member countries working to implement human-centric, safe, secure, and trustworthy AI embodied in the principles of the OECD Recommendation on AI.
Leverhulme Centre for the Future of Intelligence (CFI)
Interdisciplinary research center at the University of Cambridge exploring the nature, ethics, and impact of AI.
Median Group
Research nonprofit working on models of past and future progress in AI, intelligence enhancement, and sociology related to existential risks.
Narrow Path
A series of proposals developed by ControlAI intended for action by policymakers in order for humanity to survive artifical superintelligence.
Partnership on AI (PAI)
Convening academic, civil society, industry, and media organizations to create solutions so that AI advances positive outcomes for people and society.
Rethink Priorities
Researching solutions and strategies and mobilizing resources across various cause areas – including AI safety – in order to safeguard a flourishing present and future.
The Compendium
Living document aiming to present a coherent worldview explaining the race to AGI and extinction risks and what to do about them – in a way that is accessible to non-technical readers.
AI Governance and Safety Institute (AIGSI)
Aiming to improve institutional response to existential risk from AI by conducting research and outreach, and developing educational materials.
Strategy, Advocacy
Convergence Analysis
Building a foundational series of sociotechnical reports on key AI scenarios and governance recommendations, and conducting awareness campaigns to the general public.
AI Impacts
Answering decision-relevant questions about the future of AI, including through research, a wiki, and expert surveys. Run by MIRI.
Strategy, Forecasting
Center for Long-Term Resilience (CLTR)
Think tank aiming to transform global resilience to extreme risks by improving relevant governance, processes, and decision-making.
Strategy, Governance
Intelligence Rising
Workshop letting decision-makers experience the tensions and risks that can emerge in the highly competitive environment of AI development through an educational roleplay game.
Strategy, Governance, Training and education
AI Safety Asia (AISA)
Convening stakeholders, training civil servants and civil society leaders, and conducting regional AI policy research in Asia.
Training and education
AI Safety Camp (AISC)
3-month part-time online research program with mentorship, aimed at helping people who want to work on AI safety team up together on concrete projects.
AI Safety Hungary
Supports students and professionals in contributing to the safe development of AI, including through introductory courses.
AI Safety Initiative at Georgia Tech (AISI)
Georgia Tech community conducting training programs and research projects investigating open problems in AI safety.
AI Safety, Ethics and Society (AISES)
Course from the Center for AI Safety (CAIS) covering a wide range of risks while leveraging concepts and frameworks from existing research fields to analyze AI safety.
AISafety.com: Self-study
Comprehensive, up-to-date directory of AI safety curricula and reading lists for self-led learning at all levels.
AISafety.info
Accessible guide to AI safety for those new to the space, in the form of a comprehensive FAQ and AI safety chatbot. Project of Rob Miles.
Alignment Research Engineer Accelerator (ARENA)
4–5 week ML engineering upskilling program, focusing on alignment. Aims to provide individuals with the skills, community, and confidence to contribute directly to technical AI safety.
Apart Sprints
Short hackathons and challenges, both online and in-person around the world, focused on important questions in AI safety.
Athena Mentorship Program for Women
10-week remote mentorship program for women looking to strengthen their research skills and network in technical Al alignment research.
BlueDot Impact
Runs the standard introductory courses for people new to AI safety. Two main streams: Technical AI Safety and AI Governance.
Cambridge Boston Alignment Initiative (CBAI)
Helping students get into AI safety research via upskilling programs and fellowships. Supports AISST and MAIA.
Cambridge ERA:AI Fellowship
In-person, paid, 8-week summer research fellowship at the University of Cambridge for aspiring AI safety and governance researchers.
Center for Human-Compatible AI (CHAI): Internship
Designed for students and professionals who are interested in research in human-compatible AI. Interns work on a research project supervised by a mentor.
Center on Long-Term Risk (CLR): Fellowship
2-3 month summer research fellowship in London working on challenging research questions relevant to reducing suffering in the long-term future.
Cooperative AI Summer School
Annual program providing students and early-career professionals in AI, computer science, and related disciplines with a firm grounding in the emerging field of cooperative AI.
Foresight Fellowship
1-year program catalyzing collaboration among young scientists, engineers, and innovators working to advance technologies for the benefit of life.
Future Impact Group (FIG) Fellowship
Remote, part-time research opportunities in AI policy and philosophy for safe AI. Provides ongoing support, including coworking sessions, issue troubleshooting, and career guidance.
Global Challenges Project (GCP)
Intensive 3-day workshops for students to explore the foundational arguments around risks from advanced AI (and biotechnology). Run by Kairos.
GovAI Fellowship
3-month program by the Centre for the Governance of AI, designed to help professionals transition to working on AI governance.
Human-aligned AI Summer School
Annual 4-day program held in Prague, Czech Republic, teaching alignment research methodology through talks, workshops, and discussions.
IAPS AI Policy Fellowship
Fully-funded, 3-month program by the Institute for AI Policy and Strategy (IAPS) for professionals seeking to strengthen practical policy skills for managing the challenges of advanced AI.
Impact Academy: Global AI Safety Fellowship
Fully-funded research program connecting exceptional STEM researchers with full-time placement opportunities at AI safety organizations for up to 6 months.
Impact Research Groups (IRG)
8-week mentored research program for London students, where teams explore a research question tackling global challenges – including AI safety.
We're a global team of volunteers and professionals from various disciplines who believe AI poses a grave risk of extinction to humanity.
Help us out
Newsletters