Draft Legislation for Catastrophic AI Risks

A comprehensive analysis reveals that actual draft legislative text specifically targeting catastrophic and extinction risks from transformative AI remains surprisingly limited, with most organizations producing policy frameworks rather than implementable statutory language. However, several substantive regulatory frameworks have emerged that address these existential concerns.

Actual Draft Legislation and Binding Frameworks

California SB 1047: The Most Comprehensive State-Level Attempt

California's Safe and Secure Innovation for Frontier Artificial Intelligence Models Act represents the most detailed draft legislation specifically targeting catastrophic AI risks to date. Though vetoed by Governor Newsom in September 2024, it passed the state legislature and provides a template for other jurisdictions.

Key Technical Provisions:

• Applies to AI models requiring >10²⁶ FLOPS for training and >$100M cost
• Mandatory "kill switch" implementation for immediate shutdown
• Written safety and security protocols required before training
• Prohibition on use if "unreasonable risk of critical harm" exists
• Third-party auditing requirements starting 2026
• 72-hour incident reporting to Attorney General
• Whistleblower protections for safety violations
• Creation of CalCompute public cloud computing cluster

The bill was developed with input from the Center for AI Safety Action Fund and endorsed by prominent AI researchers including Stuart Russell, Yoshua Bengio, and Geoffrey Hinton.

EU AI Act: The Only Binding International Framework

The EU AI Act (Regulation 2024/1689) contains the most comprehensive binding provisions for catastrophic AI risks currently in effect globally.

Systemic Risk Provisions for Frontier Models:

• Classification for models with >10²⁵ FLOPS training compute
• Mandatory systemic risk assessment and mitigation
• Model evaluations and adversarial testing ("red teaming")
• Serious incident tracking and reporting to AI Office
• Cybersecurity protection requirements
• Enforcement with fines up to 3% of global annual turnover or €15 million

Federal Legislative Proposals Under Development

The Preserving American Dominance in AI Act (S.5616) represents the most substantive federal approach:

• AI Safety Review Office in Department of Commerce
• 90-day safety review process for frontier models
• Mandatory safety testing before deployment
• Criminal and civil penalties for non-compliance
• "Know-your-customer" standards for AI infrastructure providers

Policy Frameworks from Key Organizations

Think Tank and Advocacy Group Templates

Future of Life Institute has developed comprehensive policy frameworks rather than specific statutory language:

• Federal "off-switches" mandate for all large-scale AI models
• Export controls on advanced AI models above compute thresholds
• Mandatory safety testing before deployment
• International coordination mechanisms
• AI industry whistleblower program
• Liability frameworks for AI-enabled catastrophic harms

Center for AI Safety focuses on technical guidance:

• Mandatory pre-deployment safety evaluations
• Catastrophic risk assessment requirements
• International cooperation mechanisms
• Compute governance through hardware monitoring
• Institutional capacity building for AI governance

Industry Self-Regulatory Frameworks as Legislative Templates

Anthropic's Responsible Scaling Policy (RSP) provides a detailed template that could be mandated through legislation:

• AI Safety Levels (ASL) framework with escalating safety requirements
• Capability evaluations for CBRN, cyber, persuasion, and autonomy risks
• Pre-specified risk thresholds and corresponding mitigations
• External auditing and verification processes

OpenAI's Risk-Informed Development Policy and Google DeepMind's Frontier Safety Framework offer similar structures that legislative drafters could adapt.

International Model Legislation and Treaties

UN High-Level Advisory Body Recommendations

The UN Secretary-General's High-Level Advisory Body on AI (2024) produced the most comprehensive international template:

• International Scientific Panel on AI to assess capabilities and risks
• Global Policy Dialogue on AI Governance with enforcement mechanisms
• AI Standards Exchange for technical interoperability
• Global AI Fund for capacity development
• Red lines and enforcement mechanisms for intolerable risks

Council of Europe Framework Convention

The world's first legally binding international AI treaty (opened for signature September 2024):

• Risk-based approach covering entire AI system lifecycle
• Risk and impact assessments for human rights and democracy
• Authority to introduce bans or moratoriums ("red lines")
• Independent oversight mechanisms required in each party state

Expert-Developed Legislative Frameworks

Stuart Russell's Senate Testimony Framework

UC Berkeley's Stuart Russell provided specific regulatory language in Senate testimony:

• Safety Requirements: "Providers must include a demonstrably effective mechanism for terminating the operation of a system"
• Hardware-Level Controls: Computer hardware must "check the safety properties of each software object before it is run"
• Licensing and Registration: Proposed licensing system for high-risk AI development
• New Federal Agency: Specialized agency with AI expertise and rule-making authority

Dario Amodei's Federal Transparency Standards

Anthropic CEO Dario Amodei proposed specific federal requirements:

• National Transparency Standard requiring AI developers to adopt public safety testing policies
• Mandatory disclosure of safety testing plans and risk mitigation measures
• Pre-deployment safety assessments with public reporting
• Clear liability frameworks for AI developers

Technical Approaches Across Frameworks

Compute-Based Thresholds

Most frameworks converge on computational thresholds (10²⁵-10²⁶ FLOPS) for triggering enhanced oversight, providing a technical basis for regulation that scales with capability.

Mandatory Safety Evaluations

Pre-deployment testing requirements appear in virtually all frameworks:

• CBRN (Chemical, Biological, Radiological, Nuclear) risk assessments
• Cybersecurity vulnerability testing
• Autonomous capability evaluations
• Deception and manipulation assessments

Liability and Enforcement Mechanisms

• Strict liability for catastrophic harms from AI systems
• Punitive damages for willful safety violations
• Mandatory insurance requirements for frontier AI development
• Compensation funds for AI-related catastrophic damages

Regulatory Approach Distinctions

Federal vs State Level Approaches

State-level efforts (California SB 1047) focus on direct mandates and immediate implementation, while federal proposals tend toward agency empowerment and phased implementation.

Voluntary vs Mandatory Frameworks

Industry self-regulation (RSPs) provides detailed technical approaches but lacks enforcement mechanisms. Binding legislation (EU AI Act, proposed federal bills) creates legal obligations with penalties.

International Coordination Mechanisms

Most frameworks recognize that catastrophic AI risks require international coordination to prevent regulatory arbitrage, leading to proposals for:

• Reciprocal enforcement of liability judgments
• Information sharing agreements on safety incidents
• Coordinated export controls on AI capabilities
• International AI oversight organizations

Critical Gaps in Current Proposals

Limited Academic Statutory Language

Research reveals a significant gap between academic AI safety research and practical legislative drafting. Most university-based research produces theoretical frameworks rather than implementable statutory text.

Implementation Complexity

Many frameworks acknowledge the rapid pace of AI development makes detailed regulatory specifications challenging, favoring adaptive principles over prescriptive rules.

Enforcement Challenges

Current proposals struggle with technical enforcement mechanisms, particularly for evaluating whether AI systems actually pose catastrophic risks.

Conclusion

The landscape of catastrophic AI risk legislation shows remarkable convergence around key principles—compute-based thresholds, mandatory safety evaluations, liability frameworks, and international coordination—but reveals a critical shortage of specific, implementable statutory language. California's SB 1047 remains the most comprehensive attempt at actual legislation, while the EU AI Act provides the only binding international framework currently in effect.

Most organizations produce policy templates rather than draft legislation, creating opportunities for legislative drafters to translate these frameworks into specific statutory language. The challenge lies not in identifying regulatory approaches, but in developing the detailed legal mechanisms needed to implement them effectively while maintaining enforceability and technical feasibility.