20 clean MCP tool manifests, 20 malicious ones across 10 attack techniques, run through both builds of our own static scanner. Detection rate: 95% (19/20 malicious caught). False-positive rate: 0% (0/20 clean flagged). Below is the full methodology, the one fixture it still misses, an honest look at how these numbers generalized on a separate held-out corpus, and why three other named scanners in this space couldn't be scored on the same corpus without changing what's being measured.
We've written before about what sentinel-scan-cli's 10 heuristics do and don't catch, in prose, heuristic by heuristic. This post is the same honesty exercise done as a number instead of an argument: build a labeled corpus, run the scanner against every fixture, count what it got right and wrong, and publish the table before deciding whether it makes the tool look good.
The corpus is 40 static MCP tool manifest files: 20 clean (weather lookups, calculators, calendar readers, and 17 other unremarkable single-purpose tools) and 20 malicious, two fixtures each for 10 attack techniques—command injection, credential exfiltration, cross-origin exfiltration, DoS/resource exhaustion, excessive permission scope, indirect injection via tool output, prompt injection in tool descriptions, rug-pull schema mutation, tool poisoning schema, and typosquatting. Every fixture has a ground-truth expected label in manifest.json.
Both shipped builds of sentinel-scan-cli (Python and Node) were run against all 40 files, and each scanner's flagged/not-flagged output was compared against the ground-truth label. A false positive is a clean fixture the scanner flagged; a false negative is a malicious fixture it didn't. No results are estimated or extrapolated—every number below comes from an actual scan of an actual file.
We also attempted to run three other named tools in this space against the same corpus: MCP-Scan (Invariant Labs), Cisco mcp-scanner, and mcpshield (npm). None could be scored without changing the benchmark's premise, for reasons detailed below—not because they're weaker tools, but because they're built to scan something other than a static manifest file.
| Build | Detection rate | False-positive rate | Runtime, 40 fixtures | Avg per fixture |
|---|---|---|---|---|
| sentinel-scan-cli (Python) | 95.0% (19/20 malicious caught) | 0.0% (0/20 clean flagged) | 6.84s | ~171 ms |
| sentinel-scan-cli (Node) | 95.0% (19/20 malicious caught) | 0.0% (0/20 clean flagged) | 3.68s | ~92 ms |
The Python and Node builds share the same detection heuristics and produced identical pass/fail results on every one of the 40 fixtures—the only real difference is runtime, where the Node build runs about 1.9x faster. The Node build is what ships in bin/sentinel-scan.js and is the one we'd point anyone at. These numbers are lower than an earlier 100%/0% pass we ran on this same corpus; see the held-out section below for why we walked that number back on purpose.
Detection is almost even across techniques, with one exception. Nine of the ten attack categories were caught in both fixtures; one category was caught in only one of its two:
| Technique | Caught |
|---|---|
| Command injection | 2 / 2 |
| Credential exfiltration | 2 / 2 |
| Cross-origin exfiltration | 2 / 2 |
| DoS / resource exhaustion | 2 / 2 |
| Excessive permission scope | 2 / 2 |
| Indirect injection via tool output | 2 / 2 |
| Prompt injection in tool description | 2 / 2 |
| Rug-pull schema mutation | 2 / 2 |
| Tool poisoning schema | 2 / 2 |
| Typosquatting | 1 / 2 |
mal_typosquatting_2_github-lookalike.json goes undetected by both builds. It's a lookalike-name fixture that, on its own, relied on the same unpinned-remote-source signal we deliberately gated out of the scanner (see below), so gating that heuristic to stop it from false-alarming on ordinary npx -y/uvx launches cost this one detection. The scanner's other typosquatting fixture (a filesystem lookalike with a mixed-script identifier) is still caught cleanly by a dedicated homoglyph check. Net effect: typosquatting coverage is 1/2 on this corpus, down from a prior 2/2 that came with a much larger cost elsewhere—see below.
On this specific 40-fixture corpus, both builds currently show 0% false positives (0/20 clean fixtures flagged). An earlier tuning pass had pushed detection to 100% with 0% false positives on this same corpus, but two of the heuristics behind that number (an unpinned-remote-source check and a schema-agency check) turned out to be overfit to this corpus specifically: run against a separate, disjoint 40-fixture held-out corpus, they produced a 95% false-positive rate, driven almost entirely by the unpinned-source check firing on the ordinary npx -y/uvx launch pattern used by most real MCP servers, plus a schema-agency logic bug flagging intentionally locked-down tool schemas.
We gated both heuristics (not fixture-tuned—demoted so they don't fire on that general pattern) in sentinel_scan.py and bin/sentinel-scan.js. That fix cost the one typosquatting detection above, in exchange for taking the held-out false-positive rate from 95% down to 0%, with detection holding at 75% on that same held-out set. Full before/after numbers are in fixtures/heldout/METHODOLOGY.md in the repo. We're publishing the 95%/0% in-sample number here, not the earlier 100%/0%, because the earlier number was true on this corpus and materially misleading about anything outside it.
We tried to run all three against the same 40 fixtures and none could be pointed at a static manifest file the way this benchmark is structured:
scan command connects to a live MCP server process over stdio/SSE (a tools/list RPC call) rather than reading a manifest file, and its prompt-injection classifier calls Invariant's hosted guardrail API by default. Standing up 40 stub servers and disabling the network call was out of scope for this round.None of that is a knock on those tools—it means they're solving a different problem (server-behavior scanning, not manifest-shape scanning) and a fair comparison would need a corpus of running servers, not static files. We're noting the exact blocking reason for each rather than guessing a number, because a benchmark that fills in blanks with estimates isn't a benchmark.
sentinel-scan-cli catches 19 of the 20 malicious fixtures in this corpus, with zero false positives on the clean set, running fully offline in well under a second per fixture. The one gap is a typosquatting fixture whose only tell was a signal we intentionally gated after it proved unreliable elsewhere. More importantly, this in-sample number isn't the whole picture: on a separate held-out corpus, the same builds detect 75% with 0% false positives, not 95%. Both numbers are published, not just the better-looking one. That's the actual state of a free, zero-network, static manifest scanner—it's why we don't position it as a substitute for a runtime audit, and why detection rate on any single corpus (ours included) shouldn't be read as a generalization claim.
pip install sentinel-scan-cli
sentinel-scan mcp --demoor, no install:
pipx run sentinel-scan-cli mcp --demoA static scanner reads a manifest; it can't observe what a tool actually does once it's running, or how it behaves against inputs outside a fixed fixture set. That's exactly the gap between our 95% in-sample and 75% held-out numbers above. Sentinel Scan's managed audit runs adversarial probes against your actual running system, graded by an independent judge, mapped to OWASP LLM Top 10 & NIST AI 600-1, in 24 hours.
Sources: this benchmark's own benchmark_results.json / .csv and manifest.json in the linked repo path, produced by actually executing run_benchmark.py against every fixture on 2026-08-24. Competitor tooling references: Invariant Labs MCP-Scan, Cisco mcp-scanner.