Curated resources for securing AI/ML systems across threat modeling, adversarial ML, LLM security, governance, MLSecOps, and benchmarks.
Read this in other languages: 中文
Contributions welcome! See CONTRIBUTING for details.
- ASB Open Source Ecosystem
- Quick Start
- 1. Threat Modeling & Frameworks
- 2. Adversarial Machine Learning
- 3. LLM & GenAI Security
- 4. Privacy, Safety & Governance
- 5. MLSecOps, MLOps & Supply Chain Security
- 6. Datasets & Benchmarks
- 7. Learning Resources
- 9. Related Awesome Lists
- Contributing
- Project Status & Roadmap
- License
- ASB Security Schema - Unified JSON schema for capturing AI security telemetry and sharing alerts across systems.
- asb-secure-gateway - Reference gateway enforcing ASB schema with OPA policies for AI-native traffic.
- Map the threat landscape: skim Section 1 before picking tools or controls.
- Experiment with adversarial tooling: start with the libraries in Section 2 to understand attacker capability.
- Secure LLM applications: apply the guidance and scanners in Section 3 as you build guardrails.
- Embed governance early: use the risk and privacy references in Section 4 to keep regulators happy.
- Operationalize safeguards: treat Section 5 as your MLSecOps checklist.
- MITRE ATLAS - Tactics, techniques, and case studies for attacks on AI/ML systems.
- MITRE Adversarial ML Threat Matrix - ATT&CK-style matrix translating ML pipeline attacks into concrete techniques.
- ENISA Artificial Intelligence Threat Landscape - Comprehensive overview of AI attack surfaces, assets, and mitigations.
- CISA/NSA Guidelines for Secure AI System Development - Joint principles for designing, deploying, and monitoring AI securely.
- NCSC Patterns for Secure AI System Development - Reusable architectural patterns for securing data, models, and tooling.
- NIST AI Risk Management Framework (AI RMF 1.0) - Voluntary framework covering governance, mapping, measuring, and managing AI risk.
- NIST AI RMF Playbook - Practical implementation guidance, artifacts, and crosswalks for AI RMF adoption.
- NIST AI RMF Profile: Generative AI - Draft profile translating AI RMF tasks to GenAI-specific safeguards.
- Adversarial Robustness Toolbox (ART) - Python library for evasion, poisoning, extraction, and inference attacks plus defenses.
- CleverHans - Classic adversarial example framework for benchmarking robustness.
- Foolbox - Unified interface for fast gradient-based and decision-based attacks across DL frameworks.
- AdvBox - Attack generation across CV, NLP, and speech models with multi-framework support.
- TextAttack - NLP-focused adversarial attack, augmentation, and training library.
- AutoAttack - Parameter-free ensemble of strong white-box attacks for reliable robustness evaluation.
- Adversarial Attacks and Defences: A Survey - Deep dive on threat models, attack classes, and countermeasures for DL systems.
- Security Matters: A Survey on Adversarial Machine Learning - Taxonomy linking attacker goals with defender controls across the ML lifecycle.
- SoK: Security and Privacy in Machine Learning - Foundational SoK covering threat models, privacy risks, and defense trade-offs.
- OWASP Top 10 for Large Language Model Applications - Canonical list of LLM-specific risks and mitigations.
- OWASP LLM Top 10 Unofficial Japanese Translation - Community translation of the OWASP LLM Top 10 for Japanese teams.
- open-source-llm-scanners - Catalog of scanners and fuzzers targeting LLM misuse cases.
- garak - LLM vulnerability scanner probing for jailbreaks, leakage, and safety failures.
- LLM Guard - Input/output filtering toolkit with regex, classifiers, and secret detectors for LLM apps.
- DeepTeam - Red teaming orchestration framework for multi-agent LLM penetration testing.
- Giskard - Evaluation suite catching bias, robustness, and security issues in ML/LLM pipelines.
- cyber-security-llm-agents - AutoGen-based agents for offensive and defensive AI security tasks.
Need LLM jailbreak benchmarks? Jump to Section 6.2.
- SoK: Data Reconstruction Attacks Against Machine Learning Models - Taxonomy and benchmarks for reconstruction attacks plus measurement guidance.
- SoK: Security and Privacy Risks of Healthcare AI - Sector-specific review of threats to clinical AI deployments.
- SoK: Data Minimization in Machine Learning - Framework for applying data-minimization principles throughout ML pipelines.
- MLSecOps - Opinionated repo of processes, tooling, and templates for secure ML operations.
- Automating ML Security Checks using CI/CD - Guide for wiring poisoning, bias, and drift tests into pipelines.
- Analyzing the Security of Machine Learning Research Code - NVIDIA AI Red Team playbook for auditing ML repos and dependencies.
- ModelScan - Static and dynamic scanner for catching malicious or vulnerable model artifacts before deployment.
- ImageNet-C - Standard corruption benchmark to evaluate ML robustness to common noise patterns.
- CIFAR-10-C / CIFAR-100-C - Corruption suites for CIFAR datasets spanning 19 perturbations at five severities.
- RobustBench - Leaderboard and library for adversarially robust models plus evaluation scripts.
- JailbreakBench - Open benchmark and evaluation harness for jailbreak robustness.
- JBB-Behaviors dataset - 100 misuse behaviors for red teaming LLM outputs.
- Heuristic Red Teaming - Prompt dataset and harness for stress-testing safety policies.
- ML Security Cheat Sheet - High-level primer on attack surfaces, threat models, and mitigation patterns.
- Five Essential Machine Learning Security Papers - NCC Group commentary on must-read academic work.
- Machine Learning Security Principles (Packt) - Book covering foundational concepts and defensive controls.
- Responsible AI: Adversarial Attacks on LLMs (YouTube) - Conference talk demonstrating jailbreak techniques and mitigations.
- awesome-adversarial-machine-learning (yenchenlin)
- awesome-adversarial-machine-learning (man3kin3ko)
- Awesome AI for Security
- awesome-MLSecOps
- awesome-llm-security
- Awesome LM SSP
- Awesome LLM4Security
- Awesome LLM Safety
We welcome high-signal resources that directly improve the security of AI systems.
- Threat modeling frameworks, governance standards, and incident handling references.
- Offensive and defensive research (adversarial ML, jailbreaks, poisoning, extraction, inference, safety testing).
- Production-ready tooling, datasets, benchmarks, and red teaming harnesses.
- Tutorials, talks, and books that teach practitioners how to secure AI systems.
- Use unordered list items (
-) and keep one resource per line. - Follow the format below and keep descriptions concise and plain English.
Tools / libraries
- [Project Name](https://example.com) - One-line description of what it does and why its useful.Papers / posts / datasets
- *Paper or Post Title* - Short summary plus venue or publisher if relevant.Add new entries near related content sections to keep the list curated and deduplicated. Please also double-check that added links are live and publicly accessible.
This list is early-stage and intentionally scoped to core security primitives. Near-term goals:
- Expand domains beyond generic ML/LLM (healthcare, industrial, safety-critical control).
- Track emerging GenAI-specific benchmarks and red teaming playbooks.
- Highlight production case studies once vetted.
Issues and PRs are welcome for suggestions.
This project is released under CC0 1.0. You can copy, modify, and reuse the list without asking permission.