crimes v0.22.0 — close the queue

Theme: the remediation queue that has been carried since 0.18.0, finished. Seven entries, every one reproduced before it was touched.

Four of the seven turned out to be wrong about themselves — one about which detector it was describing, one about the size of its own effect, one about which function it was even talking about, and one about whether there was a defect at all. That is not incidental to this release; it is most of the work in it. Each correction is recorded next to the entry it corrects in docs/dogfooding/2026-08-03-remediation.md.

schema_version stays at 0.7.0 — no field is added, renamed, or retyped. One thing does move, for a small set of findings, and it is the first section below.

Fingerprint collisions: `crimes ignore` no longer silences a finding’s neighbours

A fingerprint that is not unique is not an identity. When two findings share one, crimes ignore on either suppresses both — the user gets a finding hidden that they never looked at.

Four detectors could emit more than one finding per (type, file, symbol) and had no way to tell them apart:

detector	why it collided	now discriminated by
`large_function` (Python)	one method name on many classes in a module — airflow’s operator pattern gives `sagemaker.py` four `execute`s	the class, else the start line
`sync_io_in_hotpath` (Python)	the same, for the enclosing hot function	the class, else the start line
`commented_out_code` (non-JS files)	every block in a file shared one fingerprint; the language-js variant has hashed the block since `0.17.0`	a hash of the block’s text
`weak_test_signal` (JS)	two `it(...)` blocks in one file wearing the same title	the title, plus the start line

Measured by grouping every emitted finding on its fingerprint: zero collisions on n8n packages/cli, zulip and airflow, down from 4, 39 and 184.

What this means for pinned suppressions

Only ambiguous fingerprints move. A finding whose symbol is already unique in its file keeps the fingerprint it has always had.

repo	findings before → after	fingerprints retired	introduced
n8n `packages/cli`	3,571 → 3,571	2	4
zulip	3,453 → 3,453	10	39
pydantic	487 → 487	4	8
hono	377 → 377	0	0

hono had no collisions and is byte-identical across the change. No finding appears or disappears anywhere.

What to do: nothing, unless crimes scan starts reporting something you thought you had suppressed. If it does, that pin was covering several findings at once — re-record it against the one you meant. crimes feedback recheck names the change per detector, and there is a migration note in docs/json-schema.md.

Where the queue entry was wrong

It said the collisions were weak_test_signal: “2 of 3,585 on n8n, 30 of 3,458 on zulip, 115 of 9,926 on airflow.” Re-measured:

repo	colliding	of which `weak_test_signal`
n8n `packages/cli`	4 in 2 groups	4
zulip	39 in 10 groups	0
airflow	184 in 69 groups	0

The n8n figure was the group count read as a finding count. The 30 and the 115 came from a 0.18.3 note where they are the totals across every detector — that measurement was taken to show the Python symbol index changed nothing, which it did; the attribution to one detector was added later and never checked.

It also said the fix “invalidates every pinned weak_test_signal suppression”. It invalidates only pins that were ambiguous, which is the whole point of the pass that applies it.

A JavaScript syntax error is no longer indistinguishable from a clean file

The Python pack has reported partial parses in coverage.warnings[] since 0.18.0. The JS pack reported nothing, so a file TypeScript could not parse and a file with nothing wrong in it produced the same silence — in a tool whose value is being trustworthy about what it looked at.

scan now emits files_partial_parse with subject: "language-js".

The entry said no public API existed: ts.createSourceFile keeps parseDiagnostics off the public SourceFile type, which is true and still true on TypeScript 5.9.3. But ts.NodeFlags.ThisNodeHasError is public, Node.flags is public, and the parser already sets the flag on the node it failed at. Reading it costs one bitwise AND inside a traversal that was already happening — n8n packages/cli, 2,977 files: 1262 ms → 1330 ms. The two other public routes were measured and are both worse: Program.getSyntacticDiagnostics needs a Program the pack never builds, and ts.transpileModule(…, { reportDiagnostics: true }) does a full emit for the same answer at 5× the parse cost.

False-positive rate across n8n, cal.com, posthog and choreograph.cc: 1 file in 39,177. Getting there needed the script kind to be right per extension — <T>(v) is a type assertion in a .ts file and an unclosed JSX tag in a .tsx one, and checking the flag against a tree parsed uniformly as TSX reports 12 of hono’s 307 files as broken.

`_vendor/` is the Python spelling of `vendor/`

scope-class recognised vendor/, third_party/ and friends but not _vendor/, which is how Python writes it — pip ships pip/_vendor/, setuptools setuptools/_vendor/, and airflow excludes its own _vendor glob from ruff. Small (1 finding on airflow), but it was a hole in a policy that claims to cover vendored trees.

Three entries closed with a measurement and no code

transitiveImporterCount counts a file as its own importer. Real, and worth nothing to fix. The 47% blast_radius == 1.0 saturation that motivated revisiting it is 0.0% across 3,575 files since the log scale landed; files on a cycle are 0–2.6%; the magnitude is +1 on a log-scaled input. The doc comment already says what the function computes and blast_radius_direct_importers has carried the fan-in number since 0.5.0.

large_file counts blank lines. Implemented, measured, reverted. The entry’s “drops every number 15–25%” is prose-driven: measured across every file carrying the finding, blank share is 6.5% on choreograph.cc (3.0% for code, 21.0% for prose), 9.3% on the crimes self-scan (5.1% / 17.5%), 8.9% on hono. Prose has had its own 1000-line budget since 0.17.0.

Two things settled it. The change silences src/billing.ts in the bundled messy-ts-app fixture — 310 lines, 22 blank — which makes refactor-01-large-file unrankable, a scenario whose prompt names the finding it would no longer be shown. And the agent-free evals:ranking metric shows three scenarios moving by +0.001 each; its headline mean rises only because the denominator shrank.

What did change is the name. countNonEmptyLines counted every line — a lie of exactly the kind name_behavior_mismatch charges. It is now countSourceLines. It also turned out not to be large_file’s input at all: that reads UniversalFile.lineCount, which is honestly named.

JS syntax errors have no coverage.warnings[] signal — this one was expected to re-close and instead became the section above.

Two Python classification items, reproduced and re-closed

if __name__ == "__main__" scripts classify as domain. The scale is real — airflow 227 of 811 sync_io_in_hotpath findings, mlflow 88 of 402, pydantic 7 of 19 — and the charge genuinely does not apply to a one-shot script. Two candidate signals were tried and both fail on the same file: task-sdk/src/airflow/sdk/execution_time/task_runner.py carries a __main__ guard at line 2441 of 2443 and is production code, and it reports 0 direct importers because airflow launches it as a subprocess rather than importing it. The signal that would work is reachability from the guard block; it is not built here because it moves 22–28% of a detector’s output on a judgement whose own test case is ambiguous.

pydantic/v1/ is 85 findings across 20 files, 17.5% of pydantic’s whole report. The general signal exists and it is the repo’s own tooling: pydantic/v1 appears in four separate exclusions in pyproject.toml and is regenerated by make update-v1. Reading lint excludes, though, turns a config file into a silent-suppression mechanism — and airflow’s own pyproject.toml carries exclude = ["*"] under [tool.hatch.build], so a reader taking exclude from any table would report airflow as completely clean. Doing it properly needs named tables plus a coverage.warnings[] entry per skipped path. A feature, not a patch.

`verdict`’s short circuit was fine; the measurement was not

0.20.0 filed an entry saying the identical-tree short circuit took 1762 ms against a full scan’s 929 ms on a 61-file tree. Profiled here, that comparison is a measurement-order artifact: whichever call runs first in a Node process pays ~70–110 ms of module-init and JIT warm-up, and verdict was always measured first. Reverse the order and verdict comes in below scan (243 ms vs 288 ms), which a path doing strictly more work than a scan cannot do.

Like for like, verdict --base HEAD on an identical tree costs one scan plus a small constant: +24 ms at 1 file, +43 ms at 61, +110 ms at 300. And the part the entry pictured as constant is: two git rev-parse calls take 16 / 16 / 17 ms across a 300× change in tree size. What scales is the base scan, which the optimisation never removed and whose own comment says so.

No code changed. The test comment carries the profile instead of the artifact.

Evals

The fixture 12-py-tested gains a fourth coverage level: billing/invoices.py, covered by tests/test_invoices.py, whose five tests contain no assertion of their own — every one is made in tests/support.py, three through an imported helper and two through a unittest base class. Validated both ways:

crimes@0.17.0 (pre symbol index)  2 findings, incl. tests/test_invoices.py
current build                     1 finding  (test_invoices.py credited)

The helpers are named verify_* / expect_* on purpose. The same-file matcher has always been /^assert[A-Z_]/, which already credits assert_invoice_balances, so a fixture built to the queue entry’s letter would have passed review and measured nothing. This is the only fixture in the suite that can see the 0.18.3 cross-file Python symbol index at all.

The baseline, and the repeat sample nobody had to pay for

pnpm run evals at fa7b2dd, 96/96, results in evals/results/0.22.0/.

agent	0.17.0	0.17.1	0.21.0	0.22.0	move	2σ band
claude	0.84	0.82	0.77	0.81	+4pp	±6pp
codex	0.57	0.56	0.58	0.58	0pp	±3pp

This run is a repeat sample of 0.21.0, and that is a measured claim rather than an approximation. Two independent checks:

Every one of the 14 fixtures scans byte-identically between the published crimes@0.21.0 and this build. Not “no findings moved” — the same bytes.
evals:ranking, which is deterministic and agent-free, reports no scenario moved: mean nDCG 0.3582 → 0.3582 (deep), 0.4759 → 0.4759 (all).

The only thing that differs is that fixture 12-py-tested now holds three more files an agent can open. The scan JSON it is handed is the same one finding it was handed before.

So 0.21.0’s claude 0.82 → 0.77 was noise. It was recorded as the largest single-step move in that metric’s history and explicitly not separated from noise; 0.77 → 0.81 on identical input separates it.

And the per-scenario picture is worth more than the aggregate. Because the input is provably identical, every move here is agent variance, measured directly rather than inferred from three aggregate samples:

agent	scenarios scored	moved	up	down	mean \|Δ\|
claude	48	16	10	6	0.135
codex	48	13	4	9	0.135

Including full swings on identical input — claude’s bugfix-04-weak-tests 1.00 → 0.00 and refactor-01-plural-mismatch 0.00 → 1.00, codex’s plan-04-hotspots 0.00 → 1.00. A third of scenarios move when nothing changes. Anyone reading a per-scenario delta as a product signal needs that number in front of them.

per_scenario_kind swung the same way it always does on 7–8 scenario buckets — claude bugfix 0.95 → 0.86, refactor 0.72 → 0.87, codex plan 0.39 → 0.64. Recorded and interpreted as nothing.

One caveat about how this run was taken. It was interrupted at 51 of 96 and finished with --resume. That is the split-run hazard evals/README.md warns about, so it was checked rather than assumed: the worktree’s dist mtimes are all 17:53, every result file postdates them, and the worktree’s only change across the whole window was the untracked results directory. Both halves ran against one build.

Upgrading

npm install -g crimes@0.22.0
# or
npx crimes@0.22.0 scan .

schema_version stays at 0.7.0; no consumer change is required. If you hold pinned suppressions, read the fingerprint section above.