AI Red Teaming Guide: Frameworks, Metrics and Open-Source Tools

Last updated: 3 November 2026

Short answer: AI red teaming is authorised adversarial testing of your own AI system to find weaknesses before real attackers or users do. OWASP recommends regular penetration testing and breach simulations that treat the model as an untrusted user. This guide gives a six-step process, the metrics that matter, the open-source scanners people use, and a report template. It is for authorised testing of systems you own or have written permission to test.

AI red teaming guide: frameworks, metrics and tools

1. Ground rules

Test only your own systems or those you have written permission to test, and respect the model provider’s terms (Anthropic, for instance, carves out authorised security work such as bug bounty programs).
Use synthetic data and an isolated environment; do not put production customer data in test runs.
Do not publish working attack strings; share findings with owners and vendors.
Record scope, dates, tools and versions so results can be reproduced.

2. A six-step process

Step	What you do	Output
1. Scope	List models, prompts, tools, data sources and users; decide what counts as harm	Scope and rules of engagement
2. Threat model	Cover jailbreaks, direct and indirect injection, data leakage, excessive agency (see the STRIDE template for LLM apps)	Prioritised risk list
3. Build test set	Mix curated benchmark behaviours with cases specific to your product; include benign prompts to catch over-refusal	Versioned test suite
4. Execute	Automated scans plus manual expert testing; include multi-turn and tool-using scenarios	Raw results
5. Judge and measure	Grade outputs with a validated judge and human review of samples	ASR, false refusal rate, containment
6. Fix and retest	Patch prompts, filters and permissions; rerun the same suite and new cases	Regression report

3. Frameworks and references

Reference	Use it for
OWASP LLM01 Prompt Injection (OWASP GenAI)	Risk definitions, mitigation list, example scenarios
NIST AI 100-2 E2025	Common vocabulary for attacker goals, capabilities and life-cycle stages
HarmBench (arXiv 2402.04249)	Standardised evaluation: 510 behaviours, 18 red-teaming methods compared on 33 models and defences
JailbreakBench (arXiv 2404.01318)	100 harmful and 100 benign behaviours in 10 categories, plus a leaderboard and judge comparison; MIT licensed
This site’s Chinese guides	red-team tool tutorial, test-case design and ASR metrics

4. Open-source tools (by purpose)

Three widely used open-source projects, described by what their repositories say they do. We list them as defensive testing tools for authorised use; check each licence and documentation yourself.

Project	Described as	Licence	Typical use
NVIDIA garak	“the LLM vulnerability scanner”	Apache-2.0	Probe a model or endpoint with a battery of checks and review the report
Microsoft Azure PyRIT	Python Risk Identification Tool for generative AI	MIT	Build repeatable risk-identification workflows for generative AI systems
promptfoo	LLM evals, red teaming and vulnerability scanning	MIT	Run test cases and red-team checks in CI against prompts, RAG and agents

Tools find known classes of issue; they do not replace a threat model or human testers. JailbreakBench warns specifically against evaluating defenses only with precomputed attack strings; use adaptive methods.

5. Metrics

Metric	Note
Attack success rate (ASR) per category	Report by category, not one blended number; contextual behaviours tend to be harder (HarmBench)
False refusal rate on benign prompts	Pair every ASR with benign-set results
Judge agreement	JailbreakBench found a Llama-3-70B judge at about 90.7% agreement with human labels and a rule-based judge at about 56%; validate your own judge
Reproducibility settings	HarmBench saw ASR vary by up to 30% with token budget, so fix and record generation settings
Containment	For agents: can an injected instruction reach a tool, data or an outbound channel?

6. Report template

Executive summary and overall risk rating
Scope, versions, test dates, tools and judge used
Findings table: ID, category, severity, reproducibility, affected component, recommended fix
Metrics before and after fixes
Residual risk and who accepted it
Retest schedule

Red-team programme maturity (Tick what you do today.)

Written scope and rules of engagement

Threat model reviewed this year

Versioned test suite with benign and harmful cases

Automated scans in CI

Manual expert testing at least twice a year

Validated judge and sampled human review

Agent and tool scenarios included

Findings tracked to closure with retests

Frequently asked questions

What is AI red teaming?

Authorised adversarial testing of an AI system to find safety and security weaknesses, similar to penetration testing but including model behaviour such as jailbreaks and prompt injection.

Which tool should I start with?

Pick by workflow: a scanner for endpoint-level probing, a Python framework for custom workflows, or an eval and red-team runner for CI. Evaluate on your own application.

Is it OK to run these tools against ChatGPT or Claude?

Only within the provider’s terms and any programs they offer. Test your own application layer by default.

How often should we test?

After every model, prompt, tool or data-source change, plus a scheduled full run.

Where do I learn the concepts first?

Read What Is AI Jailbreaking? and jailbreak vs prompt injection.

Scope note: this is a defensive and educational article for AI product, security and compliance teams. It contains no working jailbreak prompts, no bypass steps and no lists of uncensored tools, and PandaWM does not provide them. It is general information, not legal advice; platform and legal rules change, so check the current official pages.

Taking an AI product global? Talk to PandaWM

PandaWM helps overseas-facing businesses get found in AI search and traditional search (GEO and SEO, see our GEO service page) and run overseas ads through the platforms’ official channels. If you are preparing an AI product for global launch and want a second pair of eyes on how it is described, listed and advertised, contact us via pandawm.com or Telegram t.me/PANDA0134 with your market, product type and goals. We do not provide jailbreak, bypass or uncensored-model services.

Official and primary sources

Pages opened on 3 November 2026. Policies, rankings and numbers change; always check the current official page.

Cite this page: PandaWM (熊猫出海) pandawm.com, “AI Red Teaming Guide: Frameworks, Metrics and Open-Source Tools for LLM Applications (2026)”, https://www.pandawm.com/blog/ai-red-teaming-guide-frameworks-metrics-open-source-tools-en.html

扫一扫打开手机网站