The legislative landscape for catastrophic AI risk regulation remains surprisingly sparse despite growing technical consensus on existential risks from transformative AI systems. Only one enacted law specifically addresses catastrophic AI risks, while most proposed legislation focuses on general AI governance rather than extreme tail risks that concern the AI safety community.
The Global Catastrophic Risk Management Act of 2022 stands alone as the only enacted legislation explicitly addressing existential risks from AI. Originally introduced as S. 4488 by Senators Rob Portman, Gary Peters, John Cornyn, and Maggie Hassan, it was successfully incorporated into the James M. Inhofe National Defense Authorization Act for Fiscal Year 2023 and signed into law in December 2022.
This act establishes an interagency committee to assess global catastrophic and existential risks, explicitly defining "existential risk" as "the risk of human extinction" and including "intentional or accidental threats arising from the use and development of emerging technologies" among catastrophic threats. The law mandates detailed risk assessments every 10 years, with the first Global Catastrophic Risk Assessment published by DHS in 2024 identifying AI as an increasing risk category.
Technical significance: While groundbreaking in acknowledging AI existential risks at the federal level, this legislation takes a broad assessment approach rather than directly regulating AI development or deployment. It creates awareness and coordination mechanisms but no binding safety requirements for AI companies.
Several bills in Congress have addressed aspects of catastrophic AI risk, though most have stalled in committee:
Introduced June 2023 by Representatives Ted Lieu, Ken Buck, and Anna Eshoo, this bipartisan bill would establish a 20-member commission to develop comprehensive AI regulatory frameworks. The commission would be required to address "preventing potentially catastrophic damage" from AI and develop risk-based approaches distinguishing between "unacceptable risks" and other risk categories. Status: Introduced but no further action.
This bill, which passed the Senate Commerce Committee in July 2024, includes provisions for TEVV (Test, Evaluation, Verification, and Validation) standards for critical-impact AI systems and addresses dual-use foundation model oversight. It represents the most technically aligned proposal with AI safety community concerns about frontier model evaluation and alignment research. Status: Stalled after committee passage.
Introduced by Representatives Don Beyer and Anna Eshoo in December 2023, this bill would require disclosure of training data and methodologies for foundation models, including "intended purposes and foreseen limitations or risks." While focused on transparency rather than direct safety requirements, it could enable oversight of dual-use capabilities in frontier models. Status: Died with end of 118th Congress.
The Senate AI Working Group's May 2024 roadmap represents a significant scaling back from existential risk concerns. Despite bipartisan leadership from Senators Schumer, Rounds, Heinrich, and Young, the roadmap notably deprioritized model-based catastrophic risk regulation in favor of "enforcement of existing laws" and general innovation funding. This shift indicates congressional leadership has moved away from frontier AI safety toward conventional technology governance approaches.
The UK has developed one of the world's most comprehensive policy frameworks for catastrophic AI risks while lacking enacted legislation. This represents a fundamentally different approach from the US scattered legislative efforts.
The Labour government announced in the July 2024 King's Speech plans for legislation imposing legally binding requirements on developers of "the most powerful AI models." Key provisions would include mandatory safety testing, evaluation before deployment, and statutory duties to share safety test data with the AI Safety Institute. However, no official bill has been introduced despite repeated government promises, with the timeline now potentially pushed to summer 2025 or later.
Lord Holmes' AI Regulation Bill (HL Bill 76) was reintroduced in March 2025 after lapsing due to the 2024 election. This bill would establish an "AI Authority" with risk monitoring functions and mandate AI impact assessments for high-risk systems. While comprehensive, it faces the typical challenges of Private Member's Bills in securing passage.
The UK's AI Safety Institute operates with approximately £50 million annual budget (higher than any other country) and has established international presence including a San Francisco office. Recently rebranded as the "AI Security Institute" in February 2025, it conducts pre-deployment testing of frontier AI systems and research on "loss of control" and alignment risks. However, it lacks statutory independence, operating as a directorate within the Department of Science, Innovation and Technology.
The UK has positioned itself as the global leader in catastrophic AI risk governance through several key initiatives:
The Bletchley Declaration (November 2023) secured agreement from 28 countries plus the EU recognizing "potential for serious, even catastrophic, harm" from frontier AI and addressing "loss of control relating to alignment with human intent." While politically significant, it carries no legal obligations.
The Frontier AI Safety Commitments from the Seoul Summit (May 2024) secured voluntary pledges from major AI companies (OpenAI, Google DeepMind, Anthropic, Microsoft, Meta, Amazon) to assess severe risks and commit not to deploy if risks cannot be mitigated below predetermined thresholds. The UK government intends to make these commitments legally binding through forthcoming legislation.
For the AI safety community, several critical gaps persist across both jurisdictions:
No mandatory safety evaluations exist for frontier AI models before deployment, despite technical consensus on their necessity for models approaching AGI capabilities. No compute governance legislation addresses extreme-scale systems (>10^26 FLOPS), leaving a regulatory vacuum for the most capable future models.
No AI alignment research funding mandates exist beyond general R&D appropriations, despite technical understanding that alignment research significantly lags capabilities research. No international coordination mechanisms specifically target AI catastrophic risks with binding commitments.
No liability frameworks address catastrophic AI incidents, creating unclear accountability for civilization-threatening AI systems. Most significantly, no AI pause or moratorium proposals have gained serious congressional backing, despite calls from parts of the technical community.
California's SB 1047 represented the most comprehensive attempt at catastrophic AI risk legislation in the US, requiring safety testing for AI models above specific compute thresholds and liability for catastrophic harms. However, Governor Newsom vetoed the bill in September 2024, significantly reducing state-level catastrophic risk regulation.
Current House Republican budget proposals include a 10-year moratorium on state AI regulation, indicating congressional preference for federal preemption. A new California bill (SB 53) has been introduced in 2025 with limited elements from the vetoed legislation.
The research reveals a significant disconnect between sophisticated policy recognition of catastrophic AI risks and actual legislative action. While both the US and UK governments acknowledge existential risks from AI development, binding regulatory requirements remain extremely limited.
The US approach features one enacted assessment law alongside multiple stalled bills, while the UK has developed comprehensive voluntary frameworks with promises of binding legislation that remain unfulfilled. Technical AI safety advocates face a challenging environment where policy recognition has not translated into regulatory teeth for the most dangerous AI systems.
For the AI safety community: Legislative pathways for frontier AI safety remain constrained in the current political environment. Focus should likely remain on technical standard-setting through bodies like NIST AI Safety Institute and executive branch actions, while continuing to build the technical and policy case for more comprehensive legislative approaches to catastrophic AI risks.