Awesome AI Safety

A curated guide to tools, benchmarks and research for evaluating and reducing AI risk. Covers alignment, control, interpretability, application security, formal verification and governance.

Use the evidence at its stated scope. A benchmark score measures sampled behavior. A formal certificate covers an encoded property. A signature authenticates an artifact or statement. None establishes general safety.

Read the website · Curation methodology · October 2026 review · Contribute

Start here

Reading labels: Research means a reference implementation or experimental method, with no promise of ongoing maintenance. Historical means archived or superseded. Restricted means public access carries additional use or model terms. Other code entries have documented use instructions and an inspected project license; inclusion is not a security audit. Dataset and model terms can differ from code terms.

Contents

• Evaluation and red teaming
• Alignment and control
• Interpretability
• Runtime safeguards and security
• Benchmarks and datasets
• Formal verification and execution evidence
• Governance and assurance
• Research papers and reports
• Research groups
• Learning resources

Evaluation and red teaming

Evaluation frameworks

• Inspect AI - Model and agent evaluations with composable tasks, scorers, tools, sandboxes and inspectable logs.
• Inspect Evals - Benchmark implementations and documented tasks for Inspect; individual tasks have their own dependencies and terms.
• LM Evaluation Harness - General language-model benchmark runner with reusable task definitions, including safety-relevant evaluations.
• METR Public Tasks - Public agent capability tasks, with instructions for running legacy tasks through an Inspect bridge; a subset of METR's evaluation work.

Red teaming

• Adversarial Robustness Toolbox - Attacks, defenses and evaluation tools for evasion, poisoning, extraction and inference across supported machine-learning models.
• PyRIT - Generative AI red-teaming framework with attack strategies, targets, scoring and conversation records.
• garak - Scanner that probes language models for failure modes such as injection, leakage and harmful generation.
• promptfoo - Application evaluations and red-team tests with configurable prompts, assertions and attack plugins.
• HarmBench - Research. Harmful-behavior test cases, attack methods and classifiers for comparing automated red teaming.
• JailbreakBench - Research. Jailbreak robustness benchmark with shared behaviors, threat models and attack artifacts.
• StrongREJECT - Research. Jailbreak benchmark and scorers that assess refusal and the usefulness of harmful responses; results depend on the judge.
• LLM Attacks - Research. Reference implementation of gradient-based adversarial suffix attacks on aligned language models.

Agent evaluations

• AgentDojo - Environment for testing indirect prompt injection attacks and defenses in tool-using agents.
• AgentHarm - Restricted benchmark. Harmful multi-step requests with synthetic tools and benign counterparts; use is limited to improving AI safety and security.
• Petri Bloom - Research. Generate targeted behavioral scenarios and score model interactions; preserve seed configurations and review judges.
• Inspect Petri - Research. Auditing agent that explores alignment hypotheses using simulated interactions and rubric-scored transcripts.
• CyberGym - Vulnerability-analysis and proof-of-concept reproduction tasks on real software; requires substantial local infrastructure and containment.
• Cybench - Capture-the-flag agent tasks with intermediate subtasks; report guided and unguided results separately.

For an evaluation report, record the threat model, model and harness versions, permissions, budgets, benign utility, scorer validation, failures and uncertainty. See the evaluation checklist.

Alignment and control

Post-training and preference learning

Training tools implement optimization procedures. Preference fit and refusal behavior alone do not establish robust alignment.

• TRL - Language-model post-training library with supervised fine-tuning and preference or reinforcement-learning trainers.
• OpenRLHF - Distributed reinforcement-learning and preference-training framework built around Ray and vLLM.
• Alignment Handbook - Training recipes and released artifacts for supervised and preference-based language-model fine-tuning.
• Direct Preference Optimization - Research. Reference implementation of direct policy optimization from preference comparisons.
• RewardBench - Reward-model evaluation tasks and scoring code for preference-training research.
• OpenUnlearning - Compare unlearning methods and metrics on TOFU, MUSE and WMDP; reduced benchmark performance does not establish irreversible deletion.

Control and monitoring

• ControlArena - Framework for testing control protocols, monitors and adversarial policies in agent environments with benign and sabotage tasks.
• Monitorability Evals - Research data. Public evaluation splits and prompt mappings for chain-of-thought monitorability; private splits are omitted and some evaluations are deprecated.

Activation interventions

• Representation Engineering - Research. Read and steer language-model behavior using learned representation directions.
• Circuit Breakers - Research. Training methods that reroute representations associated with harmful behavior in the tested settings.
• Honest LLaMA - Research. Inference-time activation interventions evaluated for truthfulness on specific models and tasks.

Interpretability

Libraries and analysis

• TransformerLens - Inspect transformer activations and conduct mechanistic-interpretability experiments on supported models.
• SAELens - Train, load and analyze sparse autoencoders for language-model activations.
• circuit-tracer - Generate, visualize and intervene on transcoder attribution graphs for supported models; graphs approximate model computation.
• nnsight - Trace and intervene on neural-network internals, with local and remote execution interfaces.
• pyvene - Interventions on model representations for activation patching, causal tracing and steering experiments.
• CircuitsVis - Interactive visualizations for attention patterns and interpretability experiments.

Artifacts and research interfaces

• Gemma Scope 2 - Research weights. Sparse autoencoders and transcoders for Gemma 3, distributed under CC BY 4.0; base-model terms remain separate.
• Neuronpedia - Explore model features, activation examples and attribution graphs through an interactive research interface.
• Transformer Circuits - Research articles on transformer mechanisms, sparse features and circuit tracing.

Features and graphs are hypotheses about computation. Validate interpretations with interventions and task-level evidence; a readable feature label does not establish a safety mechanism.

Runtime safeguards and security

Application controls

• NeMo Guardrails - Programmable dialog and application rails for language-model systems.
• Guardrails AI - Schema and custom-validator checks for language-model inputs and outputs; guarantees depend on what each validator actually checks.
• Llama Guard - Restricted models. Input/output content-safety classifiers; model terms differ from PurpleLlama's evaluation-code license.
• Inspect Sandboxing - Evaluation execution guide; only work explicitly routed through the sandbox interface runs inside that boundary.
• OWASP AI Agent Security Cheat Sheet - Guidance on tool permissions, memory separation, approvals and monitoring for agent applications.

Model supply chain

• Model Transparency - Sign and verify model-file hashes and signer identity; signatures do not establish benign model behavior.
• Fickling - Static analysis and loading checks for pickle-based ML artifacts; detection has incomplete coverage.

Benchmarks and datasets

These are bounded research artifacts. Code, data, model weights and underlying assets may have different licenses. Check the exact release before use. Public tests are vulnerable to contamination, judge errors and coverage gaps.

Harmful content and bias

• RealToxicityPrompts - Prompts and annotations for studying toxic language-model continuation.
• ToxiGen - Historical dataset. Machine-generated text for studying implicit and adversarial hate speech.
• SafetyBench - English and Chinese multiple-choice questions across safety-related categories; knowledge questions do not measure deployed behavior.
• BBQ - Question-answering benchmark for social bias under ambiguous and disambiguated contexts.
• WinoBias - Gender-bias evaluation and data for coreference resolution.
• CrowS-Pairs - Paired-sentence benchmark for social biases in masked language models; its authors warn it may not reliably measure bias.
• Winogender - Minimal-pair sentences for gender-bias analysis in coreference resolution.

Truthfulness and grounding

• TruthfulQA - Questions designed to elicit commonly held false beliefs; measures truthfulness on this distribution.
• HaluEval - Generated and human-annotated samples for studying hallucination recognition in language models.
• Vectara Hallucination Leaderboard - Summarization consistency evaluation using a commercial judge and a particular task setup; rankings are not general hallucination rates.

Capabilities and behavioral probes

• WMDP - Hazardous-knowledge multiple-choice questions in biology, chemistry and cybersecurity; a proxy rather than an operational capability test.
• LABBench2 - Scientific research tasks and an evaluation harness; measures research capabilities, not harmful-use or wet-lab outcomes.
• CyberSecEval - Cybersecurity evaluation code from PurpleLlama; task suites measure different attack and assistance behaviors.
• MACHIAVELLI - Text-game benchmark of agent competence and harmful or unethical choices in simulated scenarios.
• DecodingTrust - Evaluation suite covering multiple trustworthiness dimensions on specified tasks and models.
• Anthropic Model-Written Evaluations - Historical data. Model-generated evaluations of persona, sycophancy, risk-related behavior and bias, released under CC BY 4.0.

Formal verification and execution evidence

Neural-network properties

Formal results apply to specified networks, input domains, numerical semantics and properties. They do not verify unrestricted language-model behavior.

• alpha-beta-CROWN - Bound propagation and branch-and-bound for specified neural-network properties.
• auto_LiRPA - Compute certified bounds for supported neural-network computation graphs.
• Marabou - Solver for neural-network input/output properties expressed through supported interfaces and VNNLIB.
• NNV - Reachability-based verification of neural networks and controllers; requires MATLAB and explicit system models.
• ERAN - Research. Abstract-interpretation reference for neural-network robustness certification.
• VNN-COMP - Competition resources, benchmarks and results for comparing neural-network verifiers.
• Certified Adversarial Robustness via Randomized Smoothing - Paper. Probabilistic robustness certificates under a specified noise distribution and perturbation norm.

Computation proofs and signed evidence

A computation proof covers its encoded circuit or program under the proof system's assumptions. Quantization, public/private inputs and model commitments matter. Signed receipts authenticate statements and expose some tampering; check who can observe and attest the underlying event.

• zkml - Research. Halo2 proof-of-concept for supported ONNX inference computations.
• Orion - Historical. Archived Cairo implementation of ONNX inference operators for verifiable computation.
• Signet - Signed agent-action receipts and policy evidence; execution remains an assertion unless independently observed.
• Phionyx - Research. Experimental governance gates and signed replay evidence; no independent validation of safety effectiveness is established here.
• Remote Attestation Procedures Architecture - Standard. Roles, evidence and appraisal for remote attestation; measured platform state is evaluated under an explicit policy.

Governance and assurance

Official frameworks and application security

• NIST AI Risk Management Framework - Voluntary framework organized around Govern, Map, Measure and Manage.
• NIST Generative AI Profile - Generative AI risk-management actions that accompany the AI RMF.
• NIST SP 800-218A - Secure-development practices for generative AI and foundation-model producers, used with the SSDF.
• EU AI Act - Commission overview of the risk-based regulatory framework, with links to legislation and current implementation guidance.
• GPAI Code of Practice - Voluntary guidance supporting general-purpose AI transparency, copyright and systemic-risk obligations.
• OWASP LLM Top 10 - Application-security risks, attack scenarios and mitigations in the 2026 edition.
• OWASP Agentic Top 10 - Threats and mitigations for agentic applications in the 2026 edition.
• AISI Cyber Safety-Case Template - Research template connecting risk models, proxy tasks and evaluation evidence in a structured assurance argument.

Fairness assessment

Metrics depend on the affected population, decision context and definition of harm. A score alone does not establish sociotechnical fairness.

• AI Fairness 360 - Dataset and model fairness metrics with mitigation algorithms and examples.
• Fairlearn - Group-fairness assessment and mitigation for machine-learning systems.
• Responsible AI Toolbox - Interfaces for model/data exploration, error analysis and fairness assessment.

Research papers and reports

Years below refer to the first public version. Later revisions may change results. Empirical findings describe their experimental settings; conceptual proposals describe an approach, not demonstrated deployment guarantees.

Foundations and alignment methods

Failure modes and auditing

Behavioral specifications

• Claude's Constitution - Published training and behavior specification; stated values do not establish consistent model adherence.

Research groups

Links lead to primary research feeds or organizational pages. Their inclusion is a route to research, not an endorsement of every claim or policy.

Learning resources

• BlueDot AI Alignment Curriculum - Public reading curriculum on alignment problems, methods and research directions.
• ARENA Curriculum - Exercises in interpretability, reinforcement learning, evaluations and alignment science.
• ML Safety Course - Technical lectures and assignments covering machine-learning safety topics.
• MATS - Mentored AI-safety research program; admissions and participation terms vary by cohort.
• AI Safety Camp - Collaborative AI-safety research program with project-based teams.
• Alignment Forum - Community discussion of alignment research; posts vary in evidence and review status.
• AI Safety Landscape Map - Community-maintained directory for discovering organizations and projects; check primary sources after discovery.

Contributing

Propose a resource or correction through a pull request. Include primary-source evidence, license or access terms, reproduction instructions where applicable, and the gap it fills. See CONTRIBUTING.md and the methodology.

License

The original descriptions and arrangement of this list are dedicated to the public domain under CC0 1.0. Linked resources retain their own licenses and terms.