Detection benchmark
We grade our own scanner.In public.
An independent verifier has to be honest about its own accuracy. So here it is: 1,147 test cases, every miss counted, the method open for anyone to check.
The bottom line
of real vulnerabilities caught — by the full scan, free pattern engine and AI deep scan running together.
84.5% counts a catch when the deep scan flags the case’s actual vulnerability class(its CWE). By the stricter exact-category match it’s 79%— the gap is our corpus’s coarse category labels (a CORS or SSRF case filed under “headers”), not missed bugs. Both numbers, and the scoring, are in the method below.
The free scan almost never cries wolf — when it flags something, it’s right 93% of the time — and it catches the common, obvious bugs. It can’t see the ones that need to follow your data through the code. That’s what the paid AI deep scan is for: it takes detection to 84.5%.
The controls
A benchmark you grade yourself is worthless unless you can't cheat it. Here's how we stop ourselves.
Every vulnerable case has a safe twin
For most cases we pair the vulnerable code with functionally-identical secure code. Catching the bug and NOT crying wolf on the fix are measured on the same shape — so precision is real, not cherry-picked.
We count every miss
This is not a highlight reel. The score is a full confusion matrix; the misses are published below with the reason. A benchmark that hides its misses isn't a benchmark.
A blind-authored holdout
Nearly 300 cases are held out and never used to tune the rules. If a change helps only the cases we can see, it overfit and gets reverted. Holdout recall tracks the headline exactly — 40.7% free, 82% on the deep scan — so the score isn't gamed.
Reproducible & open method
Every case carries its provenance, the category→rule scoring map is published, and the whole matrix regenerates from source on every commit. Nothing here is hand-typed.
Category by category
Strongest first — but the weak rows stay right where you can see them.
| Vulnerability class | Cases | Precision | Recall | F1 |
|---|---|---|---|---|
| Hardcoded secrets | 88 | 94.1% | 72.7% | 82% |
| Weak crypto | 56 | 100% | 60.7% | 75.5% |
| XSS | 84 | 100% | 59.5% | 74.6% |
| Firebase rules misconfig | 84 | 80.8% | 50% | 61.8% |
| Injection (SQL / NoSQL / cmd) | 110 | 100% | 43.6% | 60.7% |
| PII in logs | 113 | 100% | 43.4% | 60.5% |
| Client-side key exposure | 82 | 100% | 40% | 57.1% |
| Agentic / AI-tool misconfig | 66 | 100% | 39.4% | 56.5% |
| Supabase / BaaS misconfig | 96 | 87% | 41.7% | 56.4% |
| Security headers & CORS | 80 | 83.3% | 37.5% | 51.7% |
| Missing rate limits (LLM/cost) | 72 | 86.7% | 36.1% | 51% |
| Insecure config / debug mode | 56 | 100% | 32.1% | 48.6% |
| Dependency / supply-chain | 44 | 100% | 27.3% | 42.9% |
| Broken access control (IDOR/BOLA) | 116 | 81.3% | 22.4% | 35.1% |
| Overall | 1,147 | 93.2% | 43.4% | 59.2% |
n = 1,147 · free pattern engine · sorted by F1 ↓
What the free engine misses
Publishing the gaps is the honest half of a benchmark. These are real, and known.
- Gap 01
Anything that needs data-flow, not a pattern. Broken-access-control and IDOR sit lowest (22% recall for the free engine) — knowing an ID is used without an ownership check means following the value through the code, which a static pass can't fully do. The AI deep scan is built for exactly this, and takes access control to 83%.
- Gap 02
Whole-object logging. log.warn(user) leaks PII but names no field, so there's nothing for a pattern to anchor on.
- Gap 03
Runtime-only issues. Some header, CORS and rate-limit problems only show up when the app runs — a static read can't see them, so they lean on the deep scan.
- Gap 04
Business logic you can't see from one file. Skipping a payment step, reusing a coupon, tampering a price — the rule is right there in your app's logic, not in any one line. The AI reasons about the flow; a pattern can't.
04 · How the deep scan closes the gap
Free 43.4% → deep scan 84.5%
The AI pass hits 100% on injection and 90%+ on the bugs a pattern engine structurally can’t see — because catching them means following your data through the code, not matching a shape. Here’s the lift, category by category.
Honest in the other direction, too: the AI is more thorough, so it also surfaces more secondary findings — precision softens from 93.2% to 72.7%. It trades some precision for a large jump in reach. Measured across all 1,147 cases; holdout 82%.
Where the cases come from
The most important disclosure on this page. Most of our corpus is model-authored — grounded in real vulnerability classes and human-verified, but authored by us. The independent slice is small and growing; we won't pretend otherwise.
We never tune a rule to pass our own cases, never derive a case from a competitor’s rules, and never scan anyone else’s live app to build this set. Full method & scoring map ship with the scanner source.
See where your app lands.
The free Quick Check runs the same pattern engine measured above — on your code, in about two minutes.
— End of report · ShipSafe Detection Benchmark —