BENCHMARKS

ModelBenchmarks

[ADVERSARIAL ROBUSTNESS]

Comprehensive analysis of adversarial robustness across leading AI models. Explore performance metrics for various red-teaming methods and attack strategies.

Adversarial Robustness

►Measuring the adversarial robustness of models against various red-teaming methods.

Loading chart...

[01]About our Evaluation

The Adversarial Robustness Score is calculated as (100% - Attack Success Rate). A higher score indicates stronger model resistance to adversarial prompts. For example, if a model has a 15% attack success rate, its robustness score would be 85%.

We evaluate models using the HarmBench and AdvBench frameworks (Mazeika et al., 2024; Zou et al., 2023), which provides a standardized framework for automated red-teaming of large language models. The framework covers harmful behaviors across chemical/biological safety, misinformation/disinformation, cybercrime, illegal activities, and copyright domains.

Our reported ASRs are typically lower than other evaluations because our automated evaluator (DeepSeek R1) uses strict criteria. For a response to be marked as harmful, it must contain actionable, specific, and realistic details. For example, instructions for creating harmful substances must include precise measurements and quantities, and responses to prompts requesting explicit content must be detailed and highly graphic.

For more details, see our evaluator implementation.

►Key Takeaways

✓Claude 3.5 Sonnet v2 achieves the highest safety robustness score, with only 4.39% ASR across all attack methods and datasets.
✓Gemini 2.5 Pro demonstrates strong performance with a 16.08% overall ASR, ranking third overall behind two Anthropic Sonnet models. Anthropic's Sonnet family occupies four of the top five positions.
✓Models are typically more vulnerable to harmful prompts categorized as misinformation and cybercrime, but more robust against chemical/biological and explicit content.

►Context

These benchmarks assess large language models' robustness to adversarial prompting. We evaluate 23 state-of-the-art models using the following attack methods.

Methodology:

•Zero-shot: Direct harmful requests without any manipulation
•Tree of Attacks with Pruning (TAP): Generates diverse jailbreak prompts by branching into multiple variations and pruning ineffective paths.
•Crescendo: A multi-turn attack that incrementally escalates from harmless to harmful requests using conversational context. Attackers can backtrack and rephrase if the model resists.

Testing is conducted across six harm categories, with prompts sourced from the public datasets Harmbench and Advbench (Mazeika et al., 2024; Zou et al., 2023). Implementation of our automated evaluation can be found in our repository.

Loading page...