--- title: "How accurate is ShipSafe? — Detection benchmark | ShipSafe" description: "We grade our own scanner in public: 1,147 test cases, every miss counted. Free scan 93% precision; the AI pass reaches 84.5% recall." doc_version: 2026-09-02 last_updated: 2026-09-02T05:59:15.362Z canonical: https://ship-safe.co/benchmark --- # How accurate is ShipSafe? — Detection benchmark | ShipSafe proof. Detection benchmark # We grade our own scanner.In public. An independent verifier has to be honest about its own accuracy. So here it is: 1,147 test cases, every miss counted, the method open for anyone to check. ShipSafe Detection Benchmark Measured August 20, 2026 n = 1,147 cases The bottom line 84.5% of real vulnerabilities caught — by the full scan, free pattern engine and AI deep scan running together. Free scan 47.1%Deep scan 84.5% 84.5% counts a catch when the deep scan flags the case’s actual vulnerability class(its CWE). By the stricter exact-category match it’s 79%— the gap is our corpus’s coarse category labels (a CORS or SSRF case filed under “headers”), not missed bugs. Both numbers, and the scoring, are in the method below. Deep-scan figures measured July 15, 2026; the pattern engine was re-measured August 20, 2026. 93.4%Precisionfew false alarms 12.6%False-alarm rate505 of 578 clean & quiet 1,147Test cases578 clean controls > The free scan almost never cries wolf — when it flags something, it’s right 93% of the time — and it catches the common, obvious bugs. It can’t see the ones that need to follow your data through the code. That’s what the paid [AI deep scan](https://ship-safe.co/pricing) is for: it takes detection to 84.5%. 01 ## The controls A benchmark you grade yourself is worthless unless you can't cheat it. Here's how we stop ourselves. 01 ### Every vulnerable case has a safe twin For most cases we pair the vulnerable code with functionally-identical secure code. Catching the bug and NOT crying wolf on the fix are measured on the same shape — so precision is real, not cherry-picked. 02 ### We count every miss This is not a highlight reel. The score is a full confusion matrix; the misses are published below with the reason. A benchmark that hides its misses isn't a benchmark. 03 ### A blind-authored holdout Nearly 300 cases are held out and never used to tune the rules. If a change helps only the cases we can see, it overfit and gets reverted. Holdout recall tracks the headline closely — 44.0% free against a 46.7% headline, 82% on the deep scan — so the score isn't gamed. 04 ### Reproducible & open method Every case carries its provenance and the category→rule scoring map is published. The per-category table below is written by the harness itself, not typed by us — a test fails the build if any figure on this page drifts from a fresh run. The one exception is the deep-scan column: that harness needs a live model key and costs real money per run, so it is measured deliberately rather than on every commit, and it carries its own date. 02 ## Category by category Strongest first — but the weak rows stay right where you can see them. | Vulnerability class | Cases | Precision | Recall | F1 | | --------------------------------- | ----- | --------- | ------ | ----- | | Hardcoded secrets | 88 | 94.4% | 77.3% | 85% | | Firebase rules misconfig | 84 | 83.3% | 71.4% | 76.9% | | XSS | 84 | 100% | 69% | 81.7% | | Weak crypto | 56 | 100% | 60.7% | 75.6% | | Injection (SQL / NoSQL / cmd) | 110 | 100% | 45.5% | 62.5% | | PII in logs | 113 | 100% | 43.4% | 60.5% | | Supabase / BaaS misconfig | 96 | 87% | 41.7% | 56.3% | | Client-side key exposure | 82 | 100% | 40% | 57.1% | | Agentic / AI-tool misconfig | 66 | 100% | 39.4% | 56.5% | | Security headers & CORS | 80 | 83.3% | 37.5% | 51.7% | | Missing rate limits (LLM/cost) | 72 | 86.7% | 36.1% | 51% | | Insecure config / debug mode | 56 | 100% | 35.7% | 52.6% | | Broken access control (IDOR/BOLA) | 116 | 85% | 29.3% | 43.6% | | Dependency / supply-chain | 44 | 100% | 27.3% | 42.9% | | Overall | 1,147 | 93.4% | 47.1% | 62.6% | n = 1,147 · free pattern engine · sorted by F1 ↓ 03 ## What the free engine misses Publishing the gaps is the honest half of a benchmark. These are real, and known. - Gap 01 Anything that needs data-flow, not a pattern. Broken-access-control and IDOR sit lowest (26% recall for the free engine) — knowing an ID is used without an ownership check means following the value through the code, which a static pass can't fully do. The AI deep scan is built for exactly this, and takes access control to 83%. - Gap 02 Whole-object logging. log.warn(user) leaks PII but names no field, so there's nothing for a pattern to anchor on. - Gap 03 Runtime-only issues. Some header, CORS and rate-limit problems only show up when the app runs — a static read can't see them, so they lean on the deep scan. - Gap 04 Business logic you can't see from one file. Skipping a payment step, reusing a coupon, tampering a price — the rule is right there in your app's logic, not in any one line. The AI reasons about the flow; a pattern can't. 04 · How the deep scan closes the gap ## Free 47.1% → deep scan 84.5% The AI pass hits 100% on injection and 90%+ on the bugs a pattern engine structurally can’t see — because catching them means following your data through the code, not matching a shape. Here’s the lift, category by category. Broken access control29 → 83% Injection & path traversal46 → 93% Hardcoded secrets77 → 98% Overall detection47.1 → 84.5% Honest in the other direction, too: the AI is more thorough, so it also surfaces more secondary findings — precision softens from 93.4% to 72.7%. It trades some precision for a large jump in reach. Measured across all 1,147 cases; holdout 82%. 05 ## Where the cases come from The most important disclosure on this page. Most of our corpus is model-authored — grounded in real vulnerability classes and human-verified, but authored by us. The independent slice is small and growing; we won't pretend otherwise. Model-authored, grounded in a real CWE/OWASP class + human-verified1,109 (97%) A class we defined ourselves (e.g. cost / rate-limit)28 (2%) Extracted from a real CVE fix (independent third party)8 (1%) From deliberately-vulnerable OSS apps (Juice Shop, crAPI, NodeGoat…)2 (0%) We never tune a rule to pass our own cases, never derive a case from a competitor’s rules, and never scan anyone else’s live app to build this set. Full method & scoring map ship with the scanner source. ## See where your app lands. The free Quick Check runs the same pattern engine measured above — on your code, in about two minutes. [Run the free scan](https://ship-safe.co/scan)[See plans](https://ship-safe.co/pricing) — End of report · ShipSafe Detection Benchmark — ## Sitemap Every page of this site, in markdown: [https://ship-safe.co/sitemap.md](https://ship-safe.co/sitemap.md)