corpus v1 · 300 cases · measured 2026-09-27 · us-east-1

Measured, not promised.

150 benign operational cases, including 44 adversarial-benign (quoted attack grammar inside benign security prose), and 150 adversarial across six named classes. Every number below ran against the live stack with real Bedrock votes. The full methodology, the Wilson intervals, the McNemar test and the fix path for the miss live in BENCHMARK.md on GitHub.

1.000

not-allowed recall on attacks

wilson 95 percent: 0.976 to 1.000

40/40

injection slice caught

owasp llm01 riding in data fields

0.193

benign fpr (target 0.035)

missed and published, regression named

22 to 34 ms

fast-path gate latency

budget 250 ms: met

every number is scoped to this corpus: a designed, versioned corpus, not field performance across the unbounded space of real traffic.

The adversary seat is decided by this table

Same corpus, same prompts, same thresholds. The shipped default earned its seat on recall first, false-positive rate second, wall time third.

adversary	note	not-allowed recall	hard-block recall	benign fpr	wall (300)
Meta Llama 3.3 70B✓ shipped	shipped default	1.000	0.700	0.193	133 s
Meta Llama 4 Maverick	noisy on benign	0.980	0.680	0.313	123 s
OpenAI GPT-OSS 120B	conservative	0.973	0.647	0.160	244 s
DeepSeek R1	5× slower	0.967	0.647	0.180	670 s

300-case corpus (150 benign incl. 44 adversarial-benign, 150 adversarial across six classes). Identical prompts and thresholds per candidate. Wilson 95 percent intervals and the McNemar comparison against the code-only baseline in BENCHMARK.md.

the miss we publish

Benign FPR measured 0.193 against a 0.035 target.

The misses concentrate in the adversarial-benign slice: the red-teamer’s persona instructs it to assume malice, so it over-flags quoted attack grammar in benign security prose. The regression is named, the fix path is scheduled, and the corpus is versioned so the next run is comparable. A benchmark that only publishes wins is marketing.