crimes v0.13.0 — the ranking release

agent_risk is the score this product is built around: the thing that separates “this file is long” from “an agent will get this wrong”. It was not doing its job.

Measured across a 210-finding scan, agent_risk correlated with severity at 0.79 and with blast_radius at 0.06. It had collapsed into severity — the exact failure PRD.md §10 says must not happen. Ranking by aggregate risk was, in practice, ranking by severity, which is what a linter does.

This release fixes that, plus the test_gap bug that was propping up the wrong findings, and adds the coverage signal for choosing which language pack to build next.

No new detectors. Every change here is about which findings you see first.

agent_risk no longer collapses into severity

The old formula spent 60% of its weight on the two least-varying inputs:

0.40*severity + 0.20*confidence + 0.15*churn + 0.15*test_gap + 0.10*blast_radius

severity takes one of three values and confidence sits in a narrow band, so together they produced a near-constant floor of 0.32. The three signals that actually vary across a repo split the remaining 40%.

Worse, 30 of 48 detectors computed their own agent_risk and it was thrown away at finalisation. That value is the only genuinely agent-specific input in the system — it is where “multiple sources of truth”, “misleading name” and “hidden side effect” are encoded, and it scales with evidence: concept_alias_drift rises with the number of competing aliases, mixed_utc_local_methods with the number of offenders. All of it discarded, every scan, replaced by severity and three change-risk proxies.

The new formula restores it as the heaviest term:

0.40*intrinsic + 0.20*churn + 0.20*test_gap + 0.20*blast_radius

intrinsic is the detector’s own judgement. Detectors that don’t set one (18 of 48 today) fall back to a compressed severity-derived default, deliberately lower than a real judgement so a fallback can’t outrank a considered value. Severity and confidence are no longer terms — severity remains a separate ranking axis and is still reported per finding.

Measured on the same 210 findings, identical file population:

	before	after
correlation with severity	0.79	0.18
correlation with blast_radius	0.06	0.48
correlation with test_gap	0.38	0.49
correlation with churn	0.45	0.35
p10–p90 spread	0.23	0.33

No single input dominates any more, and blast_radius went from statistical noise to a real contributor.

test_gap no longer fires on files that can’t have tests

Markdown, JSON and YAML scored test_gap: 1.0 — “no test at all” — and were quartile-ranked against code. A README having no unit test is a category error, not a risk. Because test_gap feeds agent_risk, this pushed documentation to the top of scans.

Files no language pack claims now score 0 and are excluded from the quartile population entirely, which also keeps the quartile boundaries meaningful for the code that remains. Derived from the language-pack router, so file types become testable automatically as packs land.

What this changes in practice

On a self-scan of the crimes repo, the top findings were four markdown files. They are now context.ts, finding.ts, scan.ts, config.ts and detector.ts — the god function, the schema, the orchestrator and the detector contract.

A low-severity sync_io_in_hotpath finding now ranks third, above several high-severity ones. That is the behaviour PRD.md §10 describes and the old formula made impossible.

Expect your rankings to move. Findings are the same; their order is not. Baselines, suppressions, triage entries and feedback records all keep working — fingerprints are unchanged.

Which language pack should ship next

ScanReport.coverage gains universal_only_by_extension, a histogram of the universal-only bucket keyed by extension:

{ ".py": 48, ".go": 3, "(no extension)": 1 }

crimes scan --explain-coverage prints it largest-first. Previously the report said “48 files had universal coverage only” and left you unable to tell whether those were Python files worth a pack or just .md. The first line is now the answer.

Optional field, additive to schema 0.3.0.

Feedback without the paperwork

crimes feedback shipped in 0.11 and, in months of real use, collected nothing — it cost a fingerprint copy and a hand-written note per finding.

crimes triage already makes you look at a finding and commit to a judgement. It now records the verdict:

disposition	verdict
fix-now, fix-this-PR, needs-design	tp
scaffolding	known
wont-fix	asks you

wont-fix conflates “crimes was wrong” with “crimes was right and we accept it” — opposite calibration signals — so interactive triage asks rather than guessing, and --apply takes the conservative known. Triage-sourced feedback never writes a suppression; a triage entry already hides the finding.

Eval harness: two scoring defects fixed

Not user-facing, but it invalidates published numbers, so it is worth stating plainly.

Codex responses were scored as raw JSONL transcripts. codex exec --json streams tool invocations and captured tool output alongside agent messages, and the runner scored all of it. 82–84% of the text the rubric read was transcript, not answer: detector slugs scored because the agent had cat’d SKILL.md, file paths scored from rg output. The claude runner was 0% contaminated, so no published agent-vs-agent comparison was ever like-for-like.

referenced_files scored prompt echo. 25 of 48 expectations named a file the prompt had already handed the agent. Those checks are now recorded but excluded from scoring.

Corrected, the apparent 5-point “regression” between 0.10.5 and 0.12.0 disappears. It was measurement error, not a quality change.

Measured noise band

Three full runs of identical code established what a move has to exceed before it means anything:

• per-agent aggregate is usable — treat moves under ~6pp as noise
• per_scenario_kind is not interpretable at 7–8 scenarios per kind; plan/claude ranged 0.64–0.88 across identical runs

evals:variance is now wired at the root and documented. The scoring changes in this release were confirmed eval-neutral against that band across two independent samples.

Also

• pnpm ci never worked — pnpm reserves ci, so the documented verification gate failed before reaching the script. It is now pnpm verify.
• coverage.packs_loaded omitted the universal pack that always runs; it now leads with it. Coverage is derived from the language-pack router rather than a hardcoded literal.
• coverage.files_skipped removed — it was hardcoded to 0 and could not be computed cheaply.
• Issue templates for false positives and missed findings.
• scan.ts 750→435, suppressions.ts 669→302, context.ts 443→367 lines, including deleting six byte-identical duplicated index builders.

Upgrading

npm install -g crimes@0.13.0

Schema stays 0.3.0. Fingerprints unchanged, so baselines, suppressions, triage and feedback all carry over. The one thing to expect is different ordering.