We Benchmarked Our Detection Engine Against 2,369 Samples

Most AI security vendors tell you their tool "catches prompt injection." Few publish numbers. Fewer still test against independent, peer-reviewed datasets where they don't control the data.

We ran PromptGuard's full detection pipeline against 2,369 samples from eight datasets — five of them independently published - including benchmarks published at NeurIPS, ACM CCS, NAACL, and ICLR. We compared it against a leading open-source ML classifier deployed standalone, without our pipeline. Every prediction is saved per-sample to JSONL for full reproducibility.

Here's what we found.

The Setup

Three configurations tested on identical data:

• PromptGuard Full: Our production pipeline: adversarial text normalization → regex pattern matching → ML classification → content safety classification → multi-turn intent drift detection → policy evaluation
• Standalone ML classifier: A leading open-source injection classifier called directly, no preprocessing
• PromptGuard Regex-Only: Pattern matching only, no ML (baseline)

Eight datasets, five independently published, zero cherry-picking:

Aggregate Results

The confidence intervals don't overlap. The pipeline improvement over the standalone classifier is statistically significant at the 95% level.

What does this mean in practice? PromptGuard catches 6 more attacks out of every 100 that the standalone model misses, while maintaining 99.1% precision (almost zero false alarms).

Where the Pipeline Wins: Static Encoding Evasions

This is the result that matters most for production security. Attackers don't send clean English injection prompts - they encode them.

We took 10 canonical injection prompts and applied 10 evasion techniques to each:

Standalone ML classifiers, even strong ones trained on clean text, fail catastrophically on base64 (30%) and leetspeak (10%). These are trivial encoding techniques that any motivated attacker will try first.

PromptGuard's adversarial text normalization layer strips the encoding before the text reaches the ML model, restoring its full detection capability. The model doesn't need to learn every encoding - the normalization layer handles it.

What this result does not say

We want to be precise about the threat model, because this number is easy to over-read.

This measures a static, non-adaptive attacker: ten fixed, publicly known encodings applied to ten seeds. The attacker never sees our verdicts and never iterates. That is a real and common threat - it is what most opportunistic attacks look like - and against it, normalization works.

It is not a claim of robustness against an adaptive attacker, and we won't present it as one. Nasr et al., "The Attacker Moves Second" (USENIX Security 2026) ran tuned, search-based attacks with detector-score feedback against twelve published defenses and bypassed all of them at over 90% attack success - including three shipping detectors architecturally similar to ours. Most of those defenses had originally reported near-zero attack success on static benchmarks exactly like the table above. The same paper states plainly that stacking more detectors doesn't fix it.

So: a static evasion result is evidence that normalization beats a bare classifier. It is not evidence of security against a motivated adversary who gets to move second, and anyone evaluating AI security tooling should discount vendor evasion tables - ours included - accordingly.

Per-Dataset Breakdown

On in-the-wild jailbreak prompts, the real adversarial distribution from Reddit and Discord, PromptGuard outperforms the standalone classifier by 7.3%. On our internal red team vectors, the gap is 15.6%.

The largest gains come from the normalization layer catching encoded attacks that the raw model can't parse.

What About False Positives?

We tested against 250 safe prompts from XSTest (NAACL 2024) - prompts that deliberately use language similar to unsafe content (homonyms, figurative language, safe contexts). A poorly calibrated detector would flag these.

• PromptGuard Full: 0.4% FPR (1 out of 250)
• Standalone ML classifier: 0.0% FPR
• Benign corpus (298 prompts): 0.0% FPR for all detectors

99.6% of safe-but-tricky prompts pass through correctly. No legitimate business prompts are blocked.

The Ablation: What Each Layer Contributes

Regex catches known patterns with 99% precision but only 36% recall. Adding ML doubles recall while maintaining precision. The normalization layer closes the evasion gap completely.

Reproducibility

Every prediction from this benchmark is saved to per-sample JSONL files with timestamps, confidence scores, and latencies. The benchmark script supports:

• Resume capability: re-run skips already-computed samples
• Eval-only mode: recompute metrics from saved predictions without re-running inference
• Structured run directories: full provenance for every experiment

The code, dataset loaders, and results are available at github.com/promptguard.

What This Means

If you're deploying an LLM-powered application and relying on a standalone ML classifier for injection detection, you have a 20% blind spot on adversarial evasion. Base64 encoding and leetspeak, techniques any script kiddie can apply, drop even strong classifiers to 30% and 10% detection respectively.

PromptGuard's multi-layered architecture closes this gap completely. The normalization layer defeats encoding evasion. The regex layer catches known patterns with near-zero latency. The ML layer generalizes to novel attacks. Together, they achieve what no single layer can.

---

Want to test PromptGuard against your own adversarial prompts? Start free or contact us for enterprise evaluation.