Paper Complete 1 min read

Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety

Control evaluations that let the red team choose when to attack produce meaningfully lower safety estimates than evaluations that ignore attack selection.
Part of AI Safety & Mechanistic Interpretability Machine Learning
AI-safetyresearchmachine-learningshort
Published
Buried giant

AI control evaluations estimate how safe an oversight protocol is by having a red-team model attack it. Most evaluations have the red team attack indiscriminately. A scheming model would be choosier: it would pick its moments to attack and abort attempts that begin to look risky. We added both policies to agentic control evaluations and measured safety drops of 20–28 percentage points at a 1% audit budget across two environments. Evaluations that skip attack selection overstate how safe a protocol is.

I was co-first author. I worked on the core research direction from the start, designed and ran most of the experiments, and helped develop the attack-selection methodology with Tyler Tracy at Redwood Research. The project started during MARS 4.0 and continued on contract at Redwood.

Accepted at three ICML 2026 workshops (AI4GOOD, AI-WILD, and a FAGEN poster). Under review at NeurIPS 2026.

Poster for Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
Figure 1: The poster presented at ICML 2026.