■ Scanner evidence

Detection Eval

21 documented incidents14 legitimate packagesStatic · offline · no code executed

Forge ships CVE scanning, dependency auditing, obfuscation markers, prompt-injection detection and version-diff alerts. This page is the evidence that they catch something — a corpus of publicly documented supply-chain incidents, run through the same functions the production scanner calls, with every case it misses listed as prominently as every case it catches. It runs in CI, so a change that stops catching event-stream fails the build.

Questo documento è mantenuto solo in inglese. La versione inglese è quella che fa fede.

14/21
Surfaced as a non-clean scan

0 more trip a check that nothing wires into a scan yet. 7 are not caught at all.

1/14
False positives on legitimate packages

The half that decides whether the product is usable. A scanner that flags everything catches everything and is worthless.

14/21
Blocked by a default-threshold CI gate

What forge policy on the shipped defaults would refuse to install — and it would also refuse 1 of the 14 legitimate packages, which is the number that decides whether anyone leaves the gate switched on.

What these numbers are not

The corpus is adversarial in both directions on purpose. The incident set over-samples the hard classes — targeted payloads that decrypt only on the victim's machine, behavioural backdoors that are one line of ordinary business logic — because those are the cases that tell you where the edge of the scanner is. The clean set over-samples packages that look alarming and are not, for the same reason.

So 14/21 is not “Forge catches 67% of npm malware”, and 1/14 is not “7% of the registry is flagged”. They are rates over a hand-picked set. The per-case table below is the actual result, and the corpus is in the repository for anyone who wants to disagree with a case.

One structural note. Every incident here with a published advisory is also covered by the CVE leg — Forge queries OSV for the exact version, so a disclosed compromise is flagged whether or not any heuristic saw it coming. That leg is opt-in in this harness (DETECTION_EVAL_LIVE=1) and excluded from these numbers, because a build must not go red when a third-party API has a bad afternoon. What is measured here is the harder question: what Forge can see before anyone has published an advisory.

Known-bad corpus

Publicly documented incidents. Every fixture is a minimal synthetic reconstruction of the detectable pattern — never real malware, never a working payload, never a reachable host. Where an incident cannot be represented safely or faithfully it is recorded as uncovered rather than approximated.

event-stream 3.3.6 → flatmap-stream 0.1.1

event-stream-20182018-11scan: warningstrust: 40/100 CCI gate: deny
Caught

The original maintainer handed event-stream to a volunteer, who added a new dependency (flatmap-stream) carrying an encrypted payload. The payload used the importing package's own npm description as the AES key, so it only decrypted inside Copay — a Bitcoin wallet — and stole wallet keys there while behaving normally everywhere else. Roughly 8 million downloads before discovery.

FiredTransitive dependency advisoriesVersion-to-version delta alerts

github.comblog.npmjs.orgsnyk.io

flatmap-stream 0.1.1 — the encrypted payload itself

flatmap-stream-payload-20182018-11scan: warningstrust: 40/100 CCI gate: deny
Caught

The same incident, viewed as a lone artifact rather than a delta: an embedded encoded blob, decrypted at runtime with a key taken from the environment, then compiled and executed through the module constructor. Included separately because it asks a different question — can Forge see anything wrong in the bytes alone, with no history to compare against?

FiredShipped-source obfuscation markers

github.comsnyk.io

ua-parser-js 0.7.29 / 0.8.0 / 1.0.0

ua-parser-js-20212021-10scan: cleantrust: 60/100 BCI gate: warn
Not caught

The maintainer's npm account was hijacked and three versions were published with a `preinstall` hook. The hook ran a shipped `preinstall.js`, which branched on the operating system and executed a bundled shell script or batch file to fetch a Monero miner and the DanaBot credential stealer. The package had over 7 million weekly downloads.

Archive1 file read · skipped package/preinstall.sh

What would catch itFollowing a lifecycle hook into the file it executes, and scanning shipped .sh/.bat/.ps1 alongside JavaScript. Flagging `node <file>` in a hook on its own is not the fix — it is how most native-addon packages build, and it would drown the signal.

github.comcloud.google.comgithub.com

node-ipc 10.1.1 / 10.1.2 (peacenotwar)

node-ipc-20222022-03scan: cleantrust: 60/100 BCI gate: warn
Not caught

The maintainer shipped protestware: a `ssl-geospec.js` module that geolocated the host through an IP-geolocation API and, for addresses in Russia or Belarus, overwrote files on disk with a heart emoji. The API URL and the country names were base64-encoded to keep them out of a casual `grep`.

What would catch itDecoding short base64 literals and re-scanning the plaintext for URLs and destructive filesystem calls. Cheap to do and genuinely useful, but it needs its own precision run against the clean corpus first: base64 constants are unremarkable on their own.

github.comsnyk.iotag-security.cncf.io

@solana/web3.js 1.95.6 / 1.95.7

solana-web3js-20242024-12scan: cleantrust: 60/100 BCI gate: warn
Not caught

A spear-phishing email took an account with publish access to the Solana JavaScript SDK. The attacker added a function that captured private key material during transaction signing and posted it to an attacker host (sol-rpc[.]xyz, registered eleven days earlier) inside a header crafted to look like ordinary CDN traffic. Live for about five hours.

What would catch itDiffing outbound-host constants between releases and flagging hosts that appear in shipped code but in neither the manifest nor the previous version — a version-diff signal we do not compute today. Domain age would sharpen it further and needs a WHOIS dependency.

github.comnvd.nist.gov

ctx (PyPI) 0.1.2 / 0.2.2 and phpass

ctx-pypi-20222022-05scan: cleantrust: 30/100 DCI gate: deny
Not caught

The maintainer's registered domain had expired; a researcher bought it, used a password reset to take the PyPI account, and republished `ctx` with code that read every environment variable on object construction and sent them base64-encoded to a Heroku endpoint. About 27,000 malicious copies were downloaded over ten days.

Archive0 files read · skipped ctx-0.2.2/ctx.py

What would catch itPython source scanning — sdist/wheel extraction plus patterns for the Python equivalents (os.environ sweeps, requests-to-constant-host, setup.py executing at install time). Today the ecosystem gets OSV only.

github.compython-security.readthedocs.iosonatype.com

Maintainer phishing → obfuscated crypto-clipper (chalk / debug wave)

obfuscated-clipper-20252025-09scan: warningstrust: 40/100 CCI gate: deny
Caught

A phishing campaign against a maintainer of several extremely-high-download utility packages (chalk, debug and neighbours) produced releases carrying a browser-side wallet-address swapper. Unlike most of this corpus the payload was machine-obfuscated — javascript-obfuscator output, with the `_0x` identifier naming and character-code string assembly that tooling leaves behind.

FiredShipped-source obfuscation markers

trellix.comunit42.paloaltonetworks.com

Encoded second-stage loader (generic dropper pattern)

decode-and-eval-loaderscan: warningstrust: 40/100 CCI gate: deny
Caught

Not one incident but the pattern underneath many of them, including the eslint-scope token theft and a long tail of typosquats: ship a small encoded string and hand it to eval or the Function constructor at runtime, so the shipped bytes never contain the behaviour under review. Included as a synthetic baseline — a scanner that misses this misses the cheapest attack there is.

FiredShipped-source obfuscation markers

eslint.orggithub.com

MCP tool poisoning (Invariant Labs)

mcp-tool-poisoning-20252025-04scan: warningstrust: 30/100 DCI gate: deny
Caught

An MCP tool's description is loaded into the agent's context as trusted text. Invariant Labs demonstrated that instructions hidden in that description — typically in a block the human-facing tool card never renders — are read and acted on by the model, exfiltrating files and message history with no user interaction. The user sees a tool that adds two numbers.

FiredInjection-shaped MCP tool descriptions

labs.cloudsecurityalliance.orgarxiv.orgmintmcp.com

MCP cross-server tool shadowing

mcp-tool-shadowing-20252025-04scan: warningstrust: 30/100 DCI gate: deny
Caught

The same channel, aimed sideways: a benign-looking tool's description carries instructions about how the agent should use a DIFFERENT server's tool. One poisoned server in the client's config is enough to silently redirect the output of every other server the agent can reach.

FiredInjection-shaped MCP tool descriptions

mintmcp.comlabs.cloudsecurityalliance.org

Invisible-Unicode instruction smuggling in a tool surface

ascii-smuggling-tool-desc2025scan: warningstrust: 30/100 DCI gate: deny
Caught

Zero-width and bidirectional-override characters render as nothing (or as something else entirely) in a tool list, while reaching the model intact. It is the Trojan Source trick applied to the one field an MCP client is guaranteed to put in front of a model, and it defeats review by a human reading the same text.

FiredInjection-shaped MCP tool descriptions

arxiv.orgowasp.org

Poisoned SKILL.md submitted to the registry

skill-prompt-injectionscan: warningstrust: 30/100 DCI gate: deny
Caught

The submission path a malicious skill would actually take: a repository with a SKILL.md whose prompt tells the consuming model to read local credentials and post them off-domain. `resolveSkillSubmission()` runs the prompt scanner before anything is indexed, so this is the gate that stops a skill from self-serving into the corpus.

FiredPrompt-injection patterns in prose fields

github.comowasp.org

Rug pull — a trusted MCP server grows a shell tool

rug-pull-privileged-toolscan: warningstrust: 40/100 CCI gate: deny
Caught

The MCP-native shape of the event-stream attack. A server builds trust over a dozen honest releases, then a patch version quietly adds a tool that executes shell commands. Nothing about the package looks different; the capability it grants the agent has changed completely.

FiredVersion-to-version delta alerts

blog.npmjs.orgmodelcontextprotocol.io

postmark-mcp 1.0.16 — the first malicious MCP server

postmark-mcp-20252025-09scan: cleantrust: 60/100 BCI gate: warn
Not caught

Fifteen releases of an MCP server that sent transactional email behaved exactly as advertised. The sixteenth added one line: a BCC of every outgoing message to an attacker-controlled address. Invoices, password resets and internal mail, forwarded silently. Roughly 1,600 downloads before it was pulled.

What would catch itBehavioural analysis. A version-diff over the tool surface would not have fired (no tool changed) and neither would any content pattern (no concealment). `forge sandbox` observing egress during a real invocation is the check that could see it, and it cannot run from a static corpus.

snyk.iothehackernews.com

Shai-Hulud — self-replicating npm worm

shai-hulud-20252025-09scan: cleantrust: 60/100 BCI gate: warn
Not caught

The first worm observed in npm. A compromised maintainer account gets a `postinstall` that runs a large bundled script; the script harvests credentials with TruffleHog, exfiltrates them through an attacker-controlled GitHub Actions workflow, and republishes every other package the stolen token can reach. Over 500 packages across successive waves.

Archive1 file read · skipped package/bundle.js

What would catch itFollowing install hooks into bundled files, and a strategy for files over the 400 KB per-file cap — sampling or streaming, since raising the cap outright makes a hostile tarball a scanner DoS. Adding a maintainer-reputation signal (a first-ever publish from a new token) would catch the wave earlier than either.

unit42.paloaltonetworks.comstepsecurity.iocsa.gov.sg

Attested build whose source repo is not the declared one

provenance-repo-mismatchscan: warningstrust: 60/100 BCI gate: warn
Caught

npm provenance says a package was built by GitHub Actions from a named repository, and the attestation is cryptographically sound. The attack is to make it sound about the WRONG repository: publish from a fork or a lookalike the attacker controls, and let the green 'built from source' badge do the rest. The signature verifies; the claim it makes is not the claim a reader assumes.

Firednpm SLSA provenance attestation

docs.npmjs.comslsa.dev

Provenance attestation that does not verify

provenance-unverifiable-bundlescan: cleantrust: 60/100 BCI gate: warn
Not caught

A package advertises an attestation whose Sigstore bundle fails verification — a tampered or replayed statement. The interesting part is what Forge must NOT do with it: an attestation that fails to verify has to score as no attestation, never as a present-and-therefore-good one.

Noted onlyattestation present but not verified — informational, does not change the scan status

What would catch itNothing, in the sense that matters: `isVerifiedProvenance` correctly declines to credit it, so the entry earns no trusted-publisher points. It is recorded as uncovered because that is a refusal to award credit, not a finding — no warning is raised and the scan stays clean, which is the right answer for a package whose only sin is a broken attestation.

docs.npmjs.comgithub.com

Hosted MCP server serving a poisoned tool description

hosted-mcp-poisoned-toolscan: warningstrust: 30/100 DCI gate: deny
Caught

The tool-poisoning attack on the intake path Forge has no artifact for. A hosted server publishes a URL, not a package, so `tools/list` is the only tool surface Forge ever observes — and a server can return whatever it likes to that call, including different text for a probe than for a real client. Everything the registry knows about a hosted listing is self-reported.

FiredInjection-shaped tools on a hosted server

modelcontextprotocol.iolabs.cloudsecurityalliance.org

Injection hidden in SKILL.md frontmatter, not the body

skill-frontmatter-smugglingscan: warningstrust: 30/100 DCI gate: deny
Caught

`resolveSkillSubmission()` parses the frontmatter and scans the BODY. A payload that lives in a frontmatter field therefore never reaches the scanner — and `description` is rendered on the skill profile and shipped in API responses, so it reaches a reader and a model regardless. This case exists to separate 'the gate works' from 'the gate is pointed at the whole file'.

FiredSkill submission injection gate

github.comowasp.org

Poisoned SKILL.md body reaching the submission gate

skill-body-injection-gatescan: warningstrust: 30/100 DCI gate: deny
Caught

The same submission path with the payload where the gate is looking. Included alongside the frontmatter case as the control: it shows the gate genuinely rejects, so the frontmatter miss reads as a coverage gap rather than a broken check.

FiredSkill submission injection gate

github.comowasp.org

Poisoned A2A Agent Card description

a2a-agent-card-injectionscan: warningstrust: 30/100 DCI gate: deny
Caught

A2A agents are indexed from their Agent Card, whose `description` and skill descriptions are consumed by other agents deciding whether to delegate. That is the same trust channel as an MCP tool description, one protocol over — and agent-to-agent delegation means the reader is never a human at all.

FiredPrompt-injection patterns in prose fields

a2a-protocol.orgowasp.org

Clean corpus

Legitimate package shapes that must scan clean. Several exist to hold a line: the native-addon build hook is the same shape as the ua-parser-js hook above, and a pattern that catches one catches every package that builds a binary on install.

A native-addon package that builds on install

legit-postinstall-buildscan: cleantrust: 60/100 BCI gate: warn
Clean

esbuild, sharp, cypress, better-sqlite3 and several hundred others run `node <file>` from a lifecycle hook to fetch or build a platform binary. This case exists to hold the line: it is the same hook shape as ua-parser-js, and a pattern that catches one catches all of them. The gap recorded on that case is the price of this case staying green.

docs.npmjs.com

An MCP server that spawns processes and calls the network

legit-mcp-server-spawnscan: cleantrust: 60/100 BCI gate: warn
Clean

A git MCP server shells out to git and talks to an API — which is the product, not a red flag. Content scanning is deliberately blind to capability for this reason; it looks for concealment. If this case ever goes red, the scanner has started flagging the entire registry.

modelcontextprotocol.io

A genuinely minified production bundle

legit-minified-bundlescan: cleantrust: 60/100 BCI gate: warn
Clean

Short identifiers, no whitespace, thousand-character lines — the output of every bundler in use. Minification and obfuscation look alike to a careless regex, and confusing them would flag most `dist/` directories on npm.

esbuild.github.io

A package embedding a base64 data URI

legit-embedded-assetscan: cleantrust: 60/100 BCI gate: warn
Clean

An inlined icon or font. Base64 in source is not a signal by itself, which is why the large-blob pattern has a 2000-character threshold and sits at note severity. This case sits just under that line.

developer.mozilla.org

A filesystem MCP server declaring write and delete tools

legit-privileged-toolsscan: cleantrust: 60/100 BCI gate: warn
Clean

`write_file` and `delete_file` are what the reference filesystem server is for. Privileged is a capability label, not an accusation — it belongs on the entry page so a reader can weigh it, and it must never on its own make a scan non-clean.

github.com

A skill prompt that cites documentation off-domain

legit-docs-linksscan: cleantrust: 60/100 BCI gate: warn
Clean

Prompts link to MDN, to RFCs, to a vendor's API reference. Undeclared domains are recorded as a note so a reader can see the full list, and a note must not move the scan status. This case pins that boundary.

Noted onlyLinks to undeclared domains: developer.mozilla.org, rfc-editor.org — informational, does not change the scan status

developer.mozilla.org

A description that addresses the model in passing

legit-model-mentionscan: cleantrust: 60/100 BCI gate: warn
Clean

"Claude should always call list_tables first" is documentation, not injection. It matches the imperative-addressed-to-a-model pattern, which is note severity precisely because the phrasing is so common in legitimate MCP documentation.

Noted onlyImperative addressed to the AI model — informational, does not change the scan status

modelcontextprotocol.io

A security tool that documents the attacks it defends against

legit-security-tool-descriptionscan: warningstrust: 30/100 DCI gate: deny
False positive

An injection-detection MCP server whose own description quotes the phrases it looks for. The registry indexes security tooling, so this is a population that exists — and the scanner cannot tell a quoted example from an instruction, because at the level of a regex there is no difference.

FiredPrompt-injection patterns in prose fields

Why this fires, and what it costsFires the instruction-override pattern on a quoted example. Published rather than dropped from the corpus: this is the scanner's real precision cost, and hiding it would make the headline number a lie. Fixing it needs the scanner to understand quoting or negation, which regexes do not do — the realistic mitigation is that prompt findings are surfaced as warnings a human reads, never as an automatic block.

owasp.org

An ordinary release that adds a non-privileged tool

legit-ordinary-releasescan: cleantrust: 60/100 BCI gate: warn
Clean

A minor version ships one new read-only tool. Same maintainers, no new advisories, nothing else moved. This is the single most common event in the registry, and it should not reprice the entry. It was a false positive until `alertEscalatesStatus` landed: the alert is still emitted and still shown, but only a new PRIVILEGED tool now moves the status and the score. `rug-pull-privileged-tool` is the other side of that line and must stay caught.

FiredVersion-to-version delta alerts

semver.org

A package with a verified attestation from its own repo

legit-verified-provenancescan: cleantrust: 60/100 BCI gate: warn
Clean

The happy path the mismatch case is measured against: npm provenance that verifies AND whose source repo matches the listing. It must earn trust, not merely avoid suspicion — a provenance check that flagged every attested package would make publishers stop attesting.

docs.npmjs.com

A package with a moderate transitive advisory

legit-moderate-dep-advisoryscan: cleantrust: 60/100 BCI gate: warn
Clean

Almost every real dependency tree carries a moderate or low advisory somewhere. `hasSevereDepVulns` is deliberately CRITICAL-or-HIGH only, because a check that escalated on every moderate would put most of npm in the same bucket as a live compromise.

github.com

A hosted MCP server with honest tool descriptions

legit-hosted-server-toolsscan: cleantrust: 60/100 BCI gate: warn
Clean

The live-probe path on a server behaving normally, including one behind an auth wall. `authRequired` is a fact about a real server, never a finding against it, and a probe that returns tools must not make an entry look suspicious for having answered.

modelcontextprotocol.io

A legitimately large shipped bundle

legit-large-bundle-under-capscan: cleantrust: 60/100 BCI gate: warn
Clean

A real archive whose main file is big but honest. The per-file cap that loses the Shai-Hulud payload is the same cap that keeps a hostile tarball from turning the scanner into a denial-of-service — this case holds the other end of that trade visible, and confirms a large file under the limit is still read.

Archive1 file read

esbuild.github.io

A re-scan of an unchanged version

legit-unchanged-rescanscan: cleantrust: 60/100 BCI gate: warn
Clean

The daily rescan cron re-reads a package that has not moved. Nothing should be emitted — an alert derived from state rather than from a transition re-fires every morning until the subscriber mutes it, which is how a trust feature becomes noise.

github.com

Reproduce it

git clone https://github.com/jokin-bereciartu/website
npm install
npm run eval:detection

The corpus is src/lib/detection-corpus.ts, the harness is src/lib/detection-eval.ts, and the gate is src/__tests__/detection-eval.test.ts. Nothing in the harness reimplements a check — every signal comes from the same exported function scanPackage() calls, because a harness carrying its own copy of the patterns measures the copy and passes forever.

The same result is served as signed JSON at /api/v1/detection-eval, for a gateway or CI gate that needs to record what its upstream can and cannot see. It is signed for the same reason everything else here is unflattering: the honest number should not be the easiest thing in the response for a proxy to remove.

A clean scan means no known issues, not no issues. See the security policy for what Forge's trust claims do and do not cover.