crimes v0.21.0 — precision, where the false positives were

Theme: four detectors named in an outside field report as producing false positives. All four re-verified against main first, all four measured on real repositories before and after, and three of the four fixes are not the fix the report asked for — because the suggested rules did not survive contact with the files that prompted them.

That pattern is the most useful thing in this release. A complaint is evidence that something is wrong; it is not evidence about what.

`logic_in_comments` — a concept is not mentioned because its letters appear

The complaint: worst false-positive rate of any detector. Fired on five files in a repo that documents its reasoning unusually well, and none were actionable.

The cause, which neither suggested fix addressed: DOMAIN_TERMS were matched with String.includes.

file	term claimed	where it came from
`src/lib/types.ts`	`auth`	”Authored by the Curator persona”
`src/lib/job-processor.ts`	`utc`	”captured that outcome”
`apps/web/lib/buildNonce.ts` (cal.com)	`plan`	inside a longer word

Neither file has anything whatever to do with authentication or timezones. This is the 2e9b2da rule — never resolve a symbol by name alone — in its dumbest form.

Matching is now whole-word, with a closed set of inflections (-s, -ed, -ing, -ies, plus the e-drop before -ing/-ed). The allowlist is the point: it is what separates owner + s from auth + ored.

A plain word-boundary rule was tried first and was wrong — it broke a cal.com finding worth keeping (“Do not move this before authorization check”), because auth had been doing double duty as a stem. The vocabulary is now spelled out and grouped by concept, so admin and administrator also stop counting as two domain terms and inflating severity.

repo	before	after
choreograph	10	7
cal.com	11	11

cal.com’s flat 11 is not a null result and is reported flat rather than as a percentage. Two findings left (the substring matches) and two arrived — ones a spurious nearby match had been suppressing, because that check used includes too. One of them:

// If the user is not the owner of the event, new booking should be always pending.
// Otherwise, an owner rescheduling should be always accepted.
return !!(userId && originalRescheduledBookingOrganizerId === userId);

The comment says owner; the code says Organizer. That is the charge exactly, and a bug was hiding it.

`direct_date` — the example in the report was wrong

The complaint: “conflates reading time with recording it… the genuinely risky case is time used in a branch or comparison”, citing JobDetail.tsx — “9× Date.now(), 4× new Date(), and essentially all of it is display formatting”.

Opening the file found two poll timeouts:

if (Date.now() - startedAt >= VIDEO_POLL_TIMEOUT_MS) {

and the same shape again for the audio poll, plus a third comparison through a local binding. Three of the thirteen decide a branch, which is precisely the shape the report calls the valuable one.

So the narrowing as proposed would not have fixed the complaint about that file — it would have kept the finding. What was wrong is that the evidence could not say which three of the thirteen mattered, so a reader scanning a component full of formatTs() calls reasonably concluded they were all formatting. The detector invited the wrong conclusion, then got blamed for the conclusion.

DateUse gains usage: "compared" | "value", classified in the parser by walking up from the reading. Arithmetic, parentheses, .getTime() and non-null assertions are transparent, because `Date.now() - startedAt

= TIMEOUTputs the reading three nodes below the comparison. One hop through a localconst now = Date.now()` binding, because bind-then- compare is at least as common as the inline form.

Unknown resolves to compared: a wrong answer that way leaves severity where it was; the other way silently downgrades a real finding.

Evidence gains a line:

9× Date.now(), 4× new Date()
3 decide a branch or comparison (lines 642, 870, 1086); 10 only record or render the reading

Severity gains one rule: a file whose readings are all values caps at medium, however many. Thirteen new Date().toISOString() calls writing timestamp columns is a real testability cost and a real finding; it is not a poll timeout.

	before	after
choreograph `direct_date`	91	91
high	4	1

No finding is hidden, deliberately. JobDetail.tsx stays high and now names the three lines that matter.

`high_fan_in_fan_out` — the suggested rule was a no-op on its own example

The complaint: “src/lib/types.ts flagged at 33 importers. High fan-in is a shared types module’s entire job. Consider exempting modules whose exports are type-only.”

Measured before implementing: that file exports 24 interfaces and one const. An exports-are-type-only test fails on the exact file the complaint is about, and would re-arm the moment anyone added a constant to any types module.

ImportEdge.typeOnly was already in the graph and this detector had never consulted it. 32 of 33 importers write import type. Threshold 80%, not 100%, for the same reason the exports rule failed.

What moves is the judgement, not the evidence:

fan-in: 33 importers (p95 cutoff: 6, p99: 30)
32 of 33 importers take types only — a shared type module's coupling is
compile-time, and being depended on is its job

The finding stays and the count stays; only the p99 promotion is withheld. The reasoning, stated so it can be argued with: the coupling is real and compile-time. Change an interface and every importer fails to build — loudly, immediately, before anything ships. Ranking that alongside a runtime hub is what read as a category error.

choreograph: 36 → 36 findings, medium 8 → 7.

`name_behavior_mismatch` — building the thing you read through

The complaint: “getChoreoByDate() → calls createClient — flagged five times in api.ts alone… createClient() is constructing the client in order to do the read. Every data-access layer in every Next.js app has this shape.”

const supabase = await createClient()
const { data } = await supabase.from('choreograph_posts')…

Bound, then dereferenced. A create* called for its effect has no such follow-up: the return value is returned, discarded or destructured, never used as a receiver.

A shape rule, not a createClient allowlist. An allowlist is a treadmill — the next framework names it getConnection, makePool, initSupabase — and would bake one ecosystem’s vocabulary into a detector about naming in general.

The first version was too broad and the corpus caught it. Bound-and-dereferenced alone silently dropped a real finding:

const res = await fetch(url, { method: 'POST' })
const json = await res.json()

That fits the shape exactly, and a network call is a side effect whatever you do with the response. The callee now has to look like a constructor, with a regression test pinning the fetch case.

	before	after
choreograph `name_behavior_mismatch`	19	7

All 12 removals are createClient / createAdminClient. Everything kept is a genuine effect: fetch in the OAuth handler, createElement / setAttribute DOM building, insertAdjacentHTML, and three React set* state writers.

Weighed against the caveat, as the plan required

The field report ends by noting it describes a one-shot design task, not an ongoing hygiene loop, and that a detector noisy in that mode may be correctly tuned for the loop baseline and triage serve.

Two decisions came out of taking that seriously:

Nothing was turned into a filter. Every change here is evidence or severity. direct_date still reports all 91; high_fan_in_fan_out still reports all 36 with the count intact. A finding that is noise mid-task can be exactly what an audit run wants, and the way to serve both is to rank honestly rather than to hide.

logic_in_comments’s remaining admin hits were left alone. Five of choreograph’s seven survivors are the term admin, four of them in files whose path contains admin. A path-aware rule is the obvious next move, and it is a judgement call on a sample of one repository, not a bug — “Admin regen always means really re-run” in an admin route may genuinely be an unenforced rule. Four hits on one repo is not a corpus measurement, and §15 is the precedent for what happens when you act on one.

Schema

Unchanged at 0.7.0. No fingerprints move, so every .crimes/baseline.json, .crimes/suppressions.json and .crimes/triage.json entry carries over untouched.

Severities move on a handful of findings, which a baseline check --fail-on gate will notice in the lenient direction only. crimes feedback recheck carries a 0.21 note for each of the four detectors.

Eval baseline

evals/results/0.21.0/. 96/96 in 48m 28s.

agent	0.17.0	0.17.1	0.18.1	0.18.2	0.18.3	0.18.4	0.21.0	2σ band
claude	0.84	0.82	0.85	0.85	0.82	0.82	0.77	±6pp
codex	0.57	0.56	0.54	0.59	0.57	0.61	0.58	±3pp

claude is down 5pp, and that is the largest single-step move in the recorded history of this metric. It sits inside the ±6pp band, and codex’s −3pp sits exactly on its ±3pp edge. Neither is claimable as a regression, and neither is dismissed as noise here without evidence.

The short version of what follows: the agent-free ranking metric moved by −0.0012, so the scan’s ordering is where it was, and a correction is owed about a check run while landing 0.20.1 / 0.20.2 that answered the wrong question.

A correction to what was claimed while landing the changes

0.20.1 and 0.20.2 shipped without their own eval runs, on the stated grounds that “0 of 15 fixtures moved, compared on fingerprint and severity”. That comparison was run and it was accurate — and it was the wrong comparison.

The eval scorer has indexed finding evidence strings since 0.18.2, so evidence is part of what the measurement reads. direct_date gained an evidence line in this release. Measured after the fact:

	scenarios
claude scenarios whose `scan_context` changed at all	29 of 48
fixtures involved	01, 02, 04, 06

So the input the agents were scored against changed on four fixtures, not the one whose findings moved. The check that was run answered “did findings move?” when the question was “did anything the scorer reads move?”.

That is the same shape as the traps this codebase keeps recording: a check that passes on a correct-looking input while not covering the thing that mattered.

What the per-scenario diff does and does not settle

Against 0.18.4, scenario by scenario:

	count
score identical	53 of 96
score moved, on fixture 04 (findings actually changed)	4
score moved, on fixtures whose findings are identical	39

39 moved scenarios on fixtures whose findings did not change is a strong indication of agent nondeterminism — it is the artefact 0434d3b documented, where structural_pass_rate matches detector ids in free text and cannot see a correct answer phrased as prose.

But it does not settle it, because 29 of those scenarios did see a changed evidence index. The clean experiment has not been run.

The metric with no agent in it says the ranking barely moved

evals:ranking scores the scan alone — nDCG over the order the scan emitted, against each scenario’s expected findings as graded relevance labels. No agent, so no noise band: any delta is real. It is precisely the instrument for a release that re-ranks findings, and it was run:

version	mean nDCG (28 deep fixtures)
0.17.1	0.3534
0.18.1 – 0.18.4	0.3594
0.21.0	0.3582

−0.0012. About one tenth of one percent, on the measurement that cannot be moved by an agent having a bad day.

So the product’s ranking is where it was. Whatever moved structural_pass_rate by 5pp, it was not the order crimes puts findings in — which is the thing this release changed and the thing a user experiences.

That does not make the claude number nothing. It bounds it: the remaining candidates are agent nondeterminism and the scorer’s sensitivity to the changed evidence index, and both are properties of the measurement rather than of the tool.

What should still happen

Take repeat samples before treating the structural_pass_rate figure as settled:

pnpm run evals -- --label r2

Three identical-code runs at 0.12.1 are what established the ±6pp band in the first place, and a −5pp move deserves the same treatment rather than a paragraph explaining it away. Recorded as an open question. The ranking number above is why it is not a blocking one.

Run hygiene

Run from a dedicated git worktree, built once, per the rule 0.18.0 bought the hard way. This is the first run where that rule demonstrably mattered: the main tree’s dist was rebuilt at 15:44, inside the 15:35–16:23 run window, for unrelated 0.22.0 measurements. Every worktree dist was built at 15:32–15:33 and the worktree’s HEAD never moved, so the rebuild could not reach the run. Under the old single-checkout habit this baseline would have been invalid, exactly as 0.18.0’s was.

pnpm run evals:verify-scenarios reconciled all 48 scenarios against 13 fixture scans before the run started.

Verification

pnpm verify                # format:check + lint + build + typecheck + test
pnpm --filter crimes smoke # pack + install in a temp dir + run every command

2,193 tests, up from 2,132 at 0.20.0.

Each of the four fixes was written test-first and watched fail. Two were caught mid-flight by exactly that discipline: the logic_in_comments word-boundary rule broke a cal.com finding until the vocabulary was spelled out, and the name_behavior_mismatch factory rule swallowed a fetch until the callee had to look like a constructor. Both now carry regression tests naming the case.