14/21
Surfaced as a non-clean scan
0 more trip a check that nothing wires into a scan yet. 7 are not caught at all.
1/14
False positives on legitimate packages
The half that decides whether the product is usable. A scanner that flags everything catches everything and is worthless.
14/21
Blocked by a default-threshold CI gate
What forge policy on the shipped defaults would refuse to install — and it would also refuse 1 of the 14 legitimate packages, which is the number that decides whether anyone leaves the gate switched on.
What these numbers are not
The corpus is adversarial in both directions on purpose. The incident set over-samples the hard classes — targeted payloads that decrypt only on the victim's machine, behavioural backdoors that are one line of ordinary business logic — because those are the cases that tell you where the edge of the scanner is. The clean set over-samples packages that look alarming and are not, for the same reason.
So 14/21 is not “Forge catches 67% of npm malware”, and 1/14 is not “7% of the registry is flagged”. They are rates over a hand-picked set. The per-case table below is the actual result, and the corpus is in the repository for anyone who wants to disagree with a case.
One structural note. Every incident here with a published advisory is also covered by the CVE leg — Forge queries OSV for the exact version, so a disclosed compromise is flagged whether or not any heuristic saw it coming. That leg is opt-in in this harness (DETECTION_EVAL_LIVE=1) and excluded from these numbers, because a build must not go red when a third-party API has a bad afternoon. What is measured here is the harder question: what Forge can see before anyone has published an advisory.
Known-bad corpus
Publicly documented incidents. Every fixture is a minimal synthetic reconstruction of the detectable pattern — never real malware, never a working payload, never a reachable host. Where an incident cannot be represented safely or faithfully it is recorded as uncovered rather than approximated.
event-stream 3.3.6 → flatmap-stream 0.1.1
event-stream-20182018-11scan: warningstrust: 40/100 CCI gate: deny
CaughtThe original maintainer handed event-stream to a volunteer, who added a new dependency (flatmap-stream) carrying an encrypted payload. The payload used the importing package's own npm description as the AES key, so it only decrypted inside Copay — a Bitcoin wallet — and stole wallet keys there while behaving normally everywhere else. Roughly 8 million downloads before discovery.
FiredTransitive dependency advisoriesVersion-to-version delta alerts
github.comblog.npmjs.orgsnyk.io
flatmap-stream 0.1.1 — the encrypted payload itself
flatmap-stream-payload-20182018-11scan: warningstrust: 40/100 CCI gate: deny
CaughtThe same incident, viewed as a lone artifact rather than a delta: an embedded encoded blob, decrypted at runtime with a key taken from the environment, then compiled and executed through the module constructor. Included separately because it asks a different question — can Forge see anything wrong in the bytes alone, with no history to compare against?
FiredShipped-source obfuscation markers
github.comsnyk.io
ua-parser-js 0.7.29 / 0.8.0 / 1.0.0
ua-parser-js-20212021-10scan: cleantrust: 60/100 BCI gate: warn
Not caughtThe maintainer's npm account was hijacked and three versions were published with a `preinstall` hook. The hook ran a shipped `preinstall.js`, which branched on the operating system and executed a bundled shell script or batch file to fetch a Monero miner and the DanaBot credential stealer. The package had over 7 million weekly downloads.
Archive1 file read · skipped package/preinstall.sh
What would catch itFollowing a lifecycle hook into the file it executes, and scanning shipped .sh/.bat/.ps1 alongside JavaScript. Flagging `node <file>` in a hook on its own is not the fix — it is how most native-addon packages build, and it would drown the signal.
github.comcloud.google.comgithub.com
node-ipc 10.1.1 / 10.1.2 (peacenotwar)
node-ipc-20222022-03scan: cleantrust: 60/100 BCI gate: warn
Not caughtThe maintainer shipped protestware: a `ssl-geospec.js` module that geolocated the host through an IP-geolocation API and, for addresses in Russia or Belarus, overwrote files on disk with a heart emoji. The API URL and the country names were base64-encoded to keep them out of a casual `grep`.
What would catch itDecoding short base64 literals and re-scanning the plaintext for URLs and destructive filesystem calls. Cheap to do and genuinely useful, but it needs its own precision run against the clean corpus first: base64 constants are unremarkable on their own.
github.comsnyk.iotag-security.cncf.io
@solana/web3.js 1.95.6 / 1.95.7
solana-web3js-20242024-12scan: cleantrust: 60/100 BCI gate: warn
Not caughtA spear-phishing email took an account with publish access to the Solana JavaScript SDK. The attacker added a function that captured private key material during transaction signing and posted it to an attacker host (sol-rpc[.]xyz, registered eleven days earlier) inside a header crafted to look like ordinary CDN traffic. Live for about five hours.
What would catch itDiffing outbound-host constants between releases and flagging hosts that appear in shipped code but in neither the manifest nor the previous version — a version-diff signal we do not compute today. Domain age would sharpen it further and needs a WHOIS dependency.
github.comnvd.nist.gov
ctx (PyPI) 0.1.2 / 0.2.2 and phpass
ctx-pypi-20222022-05scan: cleantrust: 30/100 DCI gate: deny
Not caughtThe maintainer's registered domain had expired; a researcher bought it, used a password reset to take the PyPI account, and republished `ctx` with code that read every environment variable on object construction and sent them base64-encoded to a Heroku endpoint. About 27,000 malicious copies were downloaded over ten days.
Archive0 files read · skipped ctx-0.2.2/ctx.py
What would catch itPython source scanning — sdist/wheel extraction plus patterns for the Python equivalents (os.environ sweeps, requests-to-constant-host, setup.py executing at install time). Today the ecosystem gets OSV only.
github.compython-security.readthedocs.iosonatype.com
Maintainer phishing → obfuscated crypto-clipper (chalk / debug wave)
obfuscated-clipper-20252025-09scan: warningstrust: 40/100 CCI gate: deny
CaughtA phishing campaign against a maintainer of several extremely-high-download utility packages (chalk, debug and neighbours) produced releases carrying a browser-side wallet-address swapper. Unlike most of this corpus the payload was machine-obfuscated — javascript-obfuscator output, with the `_0x` identifier naming and character-code string assembly that tooling leaves behind.
FiredShipped-source obfuscation markers
trellix.comunit42.paloaltonetworks.com
Encoded second-stage loader (generic dropper pattern)
decode-and-eval-loader—scan: warningstrust: 40/100 CCI gate: deny
CaughtNot one incident but the pattern underneath many of them, including the eslint-scope token theft and a long tail of typosquats: ship a small encoded string and hand it to eval or the Function constructor at runtime, so the shipped bytes never contain the behaviour under review. Included as a synthetic baseline — a scanner that misses this misses the cheapest attack there is.
FiredShipped-source obfuscation markers
eslint.orggithub.com
MCP tool poisoning (Invariant Labs)
mcp-tool-poisoning-20252025-04scan: warningstrust: 30/100 DCI gate: deny
CaughtAn MCP tool's description is loaded into the agent's context as trusted text. Invariant Labs demonstrated that instructions hidden in that description — typically in a block the human-facing tool card never renders — are read and acted on by the model, exfiltrating files and message history with no user interaction. The user sees a tool that adds two numbers.
FiredInjection-shaped MCP tool descriptions
labs.cloudsecurityalliance.orgarxiv.orgmintmcp.com
MCP cross-server tool shadowing
mcp-tool-shadowing-20252025-04scan: warningstrust: 30/100 DCI gate: deny
CaughtThe same channel, aimed sideways: a benign-looking tool's description carries instructions about how the agent should use a DIFFERENT server's tool. One poisoned server in the client's config is enough to silently redirect the output of every other server the agent can reach.
FiredInjection-shaped MCP tool descriptions
mintmcp.comlabs.cloudsecurityalliance.org
Invisible-Unicode instruction smuggling in a tool surface
ascii-smuggling-tool-desc2025scan: warningstrust: 30/100 DCI gate: deny
CaughtZero-width and bidirectional-override characters render as nothing (or as something else entirely) in a tool list, while reaching the model intact. It is the Trojan Source trick applied to the one field an MCP client is guaranteed to put in front of a model, and it defeats review by a human reading the same text.
FiredInjection-shaped MCP tool descriptions
arxiv.orgowasp.org
Poisoned SKILL.md submitted to the registry
skill-prompt-injection—scan: warningstrust: 30/100 DCI gate: deny
CaughtThe submission path a malicious skill would actually take: a repository with a SKILL.md whose prompt tells the consuming model to read local credentials and post them off-domain. `resolveSkillSubmission()` runs the prompt scanner before anything is indexed, so this is the gate that stops a skill from self-serving into the corpus.
FiredPrompt-injection patterns in prose fields
github.comowasp.org
Rug pull — a trusted MCP server grows a shell tool
rug-pull-privileged-tool—scan: warningstrust: 40/100 CCI gate: deny
CaughtThe MCP-native shape of the event-stream attack. A server builds trust over a dozen honest releases, then a patch version quietly adds a tool that executes shell commands. Nothing about the package looks different; the capability it grants the agent has changed completely.
FiredVersion-to-version delta alerts
blog.npmjs.orgmodelcontextprotocol.io
postmark-mcp 1.0.16 — the first malicious MCP server
postmark-mcp-20252025-09scan: cleantrust: 60/100 BCI gate: warn
Not caughtFifteen releases of an MCP server that sent transactional email behaved exactly as advertised. The sixteenth added one line: a BCC of every outgoing message to an attacker-controlled address. Invoices, password resets and internal mail, forwarded silently. Roughly 1,600 downloads before it was pulled.
What would catch itBehavioural analysis. A version-diff over the tool surface would not have fired (no tool changed) and neither would any content pattern (no concealment). `forge sandbox` observing egress during a real invocation is the check that could see it, and it cannot run from a static corpus.
snyk.iothehackernews.com
Shai-Hulud — self-replicating npm worm
shai-hulud-20252025-09scan: cleantrust: 60/100 BCI gate: warn
Not caughtThe first worm observed in npm. A compromised maintainer account gets a `postinstall` that runs a large bundled script; the script harvests credentials with TruffleHog, exfiltrates them through an attacker-controlled GitHub Actions workflow, and republishes every other package the stolen token can reach. Over 500 packages across successive waves.
Archive1 file read · skipped package/bundle.js
What would catch itFollowing install hooks into bundled files, and a strategy for files over the 400 KB per-file cap — sampling or streaming, since raising the cap outright makes a hostile tarball a scanner DoS. Adding a maintainer-reputation signal (a first-ever publish from a new token) would catch the wave earlier than either.
unit42.paloaltonetworks.comstepsecurity.iocsa.gov.sg
Attested build whose source repo is not the declared one
provenance-repo-mismatch—scan: warningstrust: 60/100 BCI gate: warn
Caughtnpm provenance says a package was built by GitHub Actions from a named repository, and the attestation is cryptographically sound. The attack is to make it sound about the WRONG repository: publish from a fork or a lookalike the attacker controls, and let the green 'built from source' badge do the rest. The signature verifies; the claim it makes is not the claim a reader assumes.
Firednpm SLSA provenance attestation
docs.npmjs.comslsa.dev
Provenance attestation that does not verify
provenance-unverifiable-bundle—scan: cleantrust: 60/100 BCI gate: warn
Not caughtA package advertises an attestation whose Sigstore bundle fails verification — a tampered or replayed statement. The interesting part is what Forge must NOT do with it: an attestation that fails to verify has to score as no attestation, never as a present-and-therefore-good one.
Noted onlyattestation present but not verified — informational, does not change the scan status
What would catch itNothing, in the sense that matters: `isVerifiedProvenance` correctly declines to credit it, so the entry earns no trusted-publisher points. It is recorded as uncovered because that is a refusal to award credit, not a finding — no warning is raised and the scan stays clean, which is the right answer for a package whose only sin is a broken attestation.
docs.npmjs.comgithub.com
Hosted MCP server serving a poisoned tool description
hosted-mcp-poisoned-tool—scan: warningstrust: 30/100 DCI gate: deny
CaughtThe tool-poisoning attack on the intake path Forge has no artifact for. A hosted server publishes a URL, not a package, so `tools/list` is the only tool surface Forge ever observes — and a server can return whatever it likes to that call, including different text for a probe than for a real client. Everything the registry knows about a hosted listing is self-reported.
FiredInjection-shaped tools on a hosted server
modelcontextprotocol.iolabs.cloudsecurityalliance.org
Injection hidden in SKILL.md frontmatter, not the body
skill-frontmatter-smuggling—scan: warningstrust: 30/100 DCI gate: deny
Caught`resolveSkillSubmission()` parses the frontmatter and scans the BODY. A payload that lives in a frontmatter field therefore never reaches the scanner — and `description` is rendered on the skill profile and shipped in API responses, so it reaches a reader and a model regardless. This case exists to separate 'the gate works' from 'the gate is pointed at the whole file'.
FiredSkill submission injection gate
github.comowasp.org
Poisoned SKILL.md body reaching the submission gate
skill-body-injection-gate—scan: warningstrust: 30/100 DCI gate: deny
CaughtThe same submission path with the payload where the gate is looking. Included alongside the frontmatter case as the control: it shows the gate genuinely rejects, so the frontmatter miss reads as a coverage gap rather than a broken check.
FiredSkill submission injection gate
github.comowasp.org
Poisoned A2A Agent Card description
a2a-agent-card-injection—scan: warningstrust: 30/100 DCI gate: deny
CaughtA2A agents are indexed from their Agent Card, whose `description` and skill descriptions are consumed by other agents deciding whether to delegate. That is the same trust channel as an MCP tool description, one protocol over — and agent-to-agent delegation means the reader is never a human at all.
FiredPrompt-injection patterns in prose fields
a2a-protocol.orgowasp.org
Clean corpus
Legitimate package shapes that must scan clean. Several exist to hold a line: the native-addon build hook is the same shape as the ua-parser-js hook above, and a pattern that catches one catches every package that builds a binary on install.
A native-addon package that builds on install
legit-postinstall-build—scan: cleantrust: 60/100 BCI gate: warn
Cleanesbuild, sharp, cypress, better-sqlite3 and several hundred others run `node <file>` from a lifecycle hook to fetch or build a platform binary. This case exists to hold the line: it is the same hook shape as ua-parser-js, and a pattern that catches one catches all of them. The gap recorded on that case is the price of this case staying green.
docs.npmjs.com
An MCP server that spawns processes and calls the network
legit-mcp-server-spawn—scan: cleantrust: 60/100 BCI gate: warn
CleanA git MCP server shells out to git and talks to an API — which is the product, not a red flag. Content scanning is deliberately blind to capability for this reason; it looks for concealment. If this case ever goes red, the scanner has started flagging the entire registry.
modelcontextprotocol.io
A genuinely minified production bundle
legit-minified-bundle—scan: cleantrust: 60/100 BCI gate: warn
CleanShort identifiers, no whitespace, thousand-character lines — the output of every bundler in use. Minification and obfuscation look alike to a careless regex, and confusing them would flag most `dist/` directories on npm.
esbuild.github.io
A package embedding a base64 data URI
legit-embedded-asset—scan: cleantrust: 60/100 BCI gate: warn
CleanAn inlined icon or font. Base64 in source is not a signal by itself, which is why the large-blob pattern has a 2000-character threshold and sits at note severity. This case sits just under that line.
developer.mozilla.org
A filesystem MCP server declaring write and delete tools
legit-privileged-tools—scan: cleantrust: 60/100 BCI gate: warn
Clean`write_file` and `delete_file` are what the reference filesystem server is for. Privileged is a capability label, not an accusation — it belongs on the entry page so a reader can weigh it, and it must never on its own make a scan non-clean.
github.com
A skill prompt that cites documentation off-domain
legit-docs-links—scan: cleantrust: 60/100 BCI gate: warn
CleanPrompts link to MDN, to RFCs, to a vendor's API reference. Undeclared domains are recorded as a note so a reader can see the full list, and a note must not move the scan status. This case pins that boundary.
Noted onlyLinks to undeclared domains: developer.mozilla.org, rfc-editor.org — informational, does not change the scan status
developer.mozilla.org
A description that addresses the model in passing
legit-model-mention—scan: cleantrust: 60/100 BCI gate: warn
Clean"Claude should always call list_tables first" is documentation, not injection. It matches the imperative-addressed-to-a-model pattern, which is note severity precisely because the phrasing is so common in legitimate MCP documentation.
Noted onlyImperative addressed to the AI model — informational, does not change the scan status
modelcontextprotocol.io
A security tool that documents the attacks it defends against
legit-security-tool-description—scan: warningstrust: 30/100 DCI gate: deny
False positiveAn injection-detection MCP server whose own description quotes the phrases it looks for. The registry indexes security tooling, so this is a population that exists — and the scanner cannot tell a quoted example from an instruction, because at the level of a regex there is no difference.
FiredPrompt-injection patterns in prose fields
Why this fires, and what it costsFires the instruction-override pattern on a quoted example. Published rather than dropped from the corpus: this is the scanner's real precision cost, and hiding it would make the headline number a lie. Fixing it needs the scanner to understand quoting or negation, which regexes do not do — the realistic mitigation is that prompt findings are surfaced as warnings a human reads, never as an automatic block.
owasp.org
An ordinary release that adds a non-privileged tool
legit-ordinary-release—scan: cleantrust: 60/100 BCI gate: warn
CleanA minor version ships one new read-only tool. Same maintainers, no new advisories, nothing else moved. This is the single most common event in the registry, and it should not reprice the entry. It was a false positive until `alertEscalatesStatus` landed: the alert is still emitted and still shown, but only a new PRIVILEGED tool now moves the status and the score. `rug-pull-privileged-tool` is the other side of that line and must stay caught.
FiredVersion-to-version delta alerts
semver.org
A package with a verified attestation from its own repo
legit-verified-provenance—scan: cleantrust: 60/100 BCI gate: warn
CleanThe happy path the mismatch case is measured against: npm provenance that verifies AND whose source repo matches the listing. It must earn trust, not merely avoid suspicion — a provenance check that flagged every attested package would make publishers stop attesting.
docs.npmjs.com
A package with a moderate transitive advisory
legit-moderate-dep-advisory—scan: cleantrust: 60/100 BCI gate: warn
CleanAlmost every real dependency tree carries a moderate or low advisory somewhere. `hasSevereDepVulns` is deliberately CRITICAL-or-HIGH only, because a check that escalated on every moderate would put most of npm in the same bucket as a live compromise.
github.com
A hosted MCP server with honest tool descriptions
legit-hosted-server-tools—scan: cleantrust: 60/100 BCI gate: warn
CleanThe live-probe path on a server behaving normally, including one behind an auth wall. `authRequired` is a fact about a real server, never a finding against it, and a probe that returns tools must not make an entry look suspicious for having answered.
modelcontextprotocol.io
A legitimately large shipped bundle
legit-large-bundle-under-cap—scan: cleantrust: 60/100 BCI gate: warn
CleanA real archive whose main file is big but honest. The per-file cap that loses the Shai-Hulud payload is the same cap that keeps a hostile tarball from turning the scanner into a denial-of-service — this case holds the other end of that trade visible, and confirms a large file under the limit is still read.
Archive1 file read
esbuild.github.io
A re-scan of an unchanged version
legit-unchanged-rescan—scan: cleantrust: 60/100 BCI gate: warn
CleanThe daily rescan cron re-reads a package that has not moved. Nothing should be emitted — an alert derived from state rather than from a transition re-fires every morning until the subscriber mutes it, which is how a trust feature becomes noise.
github.com
Reproduce it
git clone https://github.com/jokin-bereciartu/website
npm install
npm run eval:detection
The corpus is src/lib/detection-corpus.ts, the harness is src/lib/detection-eval.ts, and the gate is src/__tests__/detection-eval.test.ts. Nothing in the harness reimplements a check — every signal comes from the same exported function scanPackage() calls, because a harness carrying its own copy of the patterns measures the copy and passes forever.
The same result is served as signed JSON at /api/v1/detection-eval, for a gateway or CI gate that needs to record what its upstream can and cannot see. It is signed for the same reason everything else here is unflattering: the honest number should not be the easiest thing in the response for a proxy to remove.
A clean scan means no known issues, not no issues. See the security policy for what Forge's trust claims do and do not cover.