Detection benchmark
We grade our own scanner.In public.
An independent verifier has to be honest about its own accuracy. So here it is: 1,147 test cases, every miss counted, the method open for anyone to check.
The bottom line
of real vulnerabilities caught — by the full scan, free pattern engine and AI deep scan running together.
84.5% counts a catch when the deep scan flags the case’s actual vulnerability class(its CWE). By the stricter exact-category match it’s 79%— the gap is our corpus’s coarse category labels (a CORS or SSRF case filed under “headers”), not missed bugs. Both numbers, and the scoring, are in the method below.
Deep-scan figures measured July 15, 2026; the pattern engine was re-measured August 7, 2026.
The free scan almost never cries wolf — when it flags something, it’s right 93% of the time — and it catches the common, obvious bugs. It can’t see the ones that need to follow your data through the code. That’s what the paid AI deep scan is for: it takes detection to 84.5%.
The controls
A benchmark you grade yourself is worthless unless you can't cheat it. Here's how we stop ourselves.
Every vulnerable case has a safe twin
For most cases we pair the vulnerable code with functionally-identical secure code. Catching the bug and NOT crying wolf on the fix are measured on the same shape — so precision is real, not cherry-picked.
We count every miss
This is not a highlight reel. The score is a full confusion matrix; the misses are published below with the reason. A benchmark that hides its misses isn't a benchmark.
A blind-authored holdout
Nearly 300 cases are held out and never used to tune the rules. If a change helps only the cases we can see, it overfit and gets reverted. Holdout recall tracks the headline closely — 44.0% free against a 46.7% headline, 82% on the deep scan — so the score isn't gamed.
Reproducible & open method
Every case carries its provenance, the category→rule scoring map is published, and the whole matrix regenerates from source on every commit. Nothing here is hand-typed.
Category by category
Strongest first — but the weak rows stay right where you can see them.
| Vulnerability class | Cases | Precision | Recall | F1 |
|---|---|---|---|---|
| Hardcoded secrets | 88 | 94.4% | 77.3% | 85% |
| Firebase rules misconfig | 84 | 83.3% | 71.4% | 76.9% |
| XSS | 84 | 100% | 69% | 81.7% |
| Weak crypto | 56 | 100% | 60.7% | 75.6% |
| Injection (SQL / NoSQL / cmd) | 110 | 100% | 45.5% | 62.5% |
| PII in logs | 113 | 100% | 43.4% | 60.5% |
| Supabase / BaaS misconfig | 96 | 87% | 41.7% | 56.3% |
| Client-side key exposure | 82 | 100% | 40% | 57.1% |
| Agentic / AI-tool misconfig | 66 | 100% | 39.4% | 56.5% |
| Security headers & CORS | 80 | 83.3% | 37.5% | 51.7% |
| Missing rate limits (LLM/cost) | 72 | 86.7% | 36.1% | 51% |
| Insecure config / debug mode | 56 | 100% | 35.7% | 52.6% |
| Dependency / supply-chain | 44 | 100% | 27.3% | 42.9% |
| Broken access control (IDOR/BOLA) | 116 | 83.3% | 25.9% | 39.5% |
| Overall | 1,147 | 93.3% | 46.7% | 62.3% |
n = 1,147 · free pattern engine · sorted by F1 ↓
What the free engine misses
Publishing the gaps is the honest half of a benchmark. These are real, and known.
- Gap 01
Anything that needs data-flow, not a pattern. Broken-access-control and IDOR sit lowest (26% recall for the free engine) — knowing an ID is used without an ownership check means following the value through the code, which a static pass can't fully do. The AI deep scan is built for exactly this, and takes access control to 83%.
- Gap 02
Whole-object logging. log.warn(user) leaks PII but names no field, so there's nothing for a pattern to anchor on.
- Gap 03
Runtime-only issues. Some header, CORS and rate-limit problems only show up when the app runs — a static read can't see them, so they lean on the deep scan.
- Gap 04
Business logic you can't see from one file. Skipping a payment step, reusing a coupon, tampering a price — the rule is right there in your app's logic, not in any one line. The AI reasons about the flow; a pattern can't.
04 · How the deep scan closes the gap
Free 46.7% → deep scan 84.5%
The AI pass hits 100% on injection and 90%+ on the bugs a pattern engine structurally can’t see — because catching them means following your data through the code, not matching a shape. Here’s the lift, category by category.
Honest in the other direction, too: the AI is more thorough, so it also surfaces more secondary findings — precision softens from 93.3% to 72.7%. It trades some precision for a large jump in reach. Measured across all 1,147 cases; holdout 82%.
Where the cases come from
The most important disclosure on this page. Most of our corpus is model-authored — grounded in real vulnerability classes and human-verified, but authored by us. The independent slice is small and growing; we won't pretend otherwise.
We never tune a rule to pass our own cases, never derive a case from a competitor’s rules, and never scan anyone else’s live app to build this set. Full method & scoring map ship with the scanner source.
See where your app lands.
The free Quick Check runs the same pattern engine measured above — on your code, in about two minutes.
— End of report · ShipSafe Detection Benchmark —