Dogfooding round: `crimes` 0.14 → 0.17 against real codebases

Date: 2026-08-02 Build under test: crimes 0.17.0 (schema_version 0.4.0), built from source. Protocol: 2026-08-02-protocol.md Friction log: 2026-08-02-log.md

The round planned in da3a894 was never executed. Four releases of detector surface had never run against code outside their own fixtures. This is that round, widened from precision calibration to a usefulness verdict.

---

The headline

Two independent blind judges, 426 findings, 9 repositories.

judge	sample	act rate
claude (blind, cards only)	276	2.5%
codex (blind, cards only)	326	9.5%

Per-repo act counts on well-maintained code, claude judge:

repo	judged	act
pydantic	50	0
zulip	50	0
p-limit	10	0
drf	50	1
hono	43	1

On five mature, heavily-reviewed codebases, 203 findings produced two things a maintainer would act on. The judges disagreed on 26% of cards, which is itself evidence that findings are ambiguous rather than that one judge was harsh.

This is not primarily a calibration problem. Underneath it sit ~30 verified defects: every one reproduced centrally, not taken from a subagent report. Several make correct findings unreadable. Four separate ones make crimes report a clean bill of health on code it never opened. One makes the documented first-run command silently disable Python.

---

1. The corpus

Cloned repos pinned by SHA in ~/crimes-dogfood/corpus/SHAS.txt. Corpus lives outside the working tree; nothing from it is committed.

repo	SHA / origin	files scanned	findings	high	scan wall
zulip	f6470bb	3588	3759	502	81.3s
buildfest	local ~/dev	1177	1556	187	26.7s
ebg	e3b7b1e local	546	759	61	6.0s
pydantic	63f2722	637	578	77	20.8s
thickeningtime	local ~/dev	298	503	40	4.2s
improbable	local ~/dev	288	488	70	7.3s
hono	51db313	390	377	22	34.6s
drf	0e25e53	237	104	10	7.2s
p-limit	df47604	8	10	0	1.3s
judged total		7169	8134	969
posthog †	96e2437	35108	38063	9177	3 h 42 m
airflow †	acb4dde	9149	17801	929	173 s
n8n †‡	fa2d8a1	~5400	9244	n/a	never finished
mlflow †	0489fb3	6692	6472	n/a	402 s
cal.com †	038381a	5051	4562	n/a	458 s
ortomate-studio †	8ad20ce	619	851	55	675 s
choreograph.cc †	local ~/dev	320	490	33	4.8 s
mysidekick-cc †	local ~/dev	239	241	25	4 s
corpus total		~58,000	~78,000

† scanned after the judging sample closed; counted in §3 and §5, and in the zero-firing analysis, but never in an act-rate table. ‡ n8n’s whole-repo scan never completed (§3.26); its figures are the union of four subtree scans and are not equivalent to a full run.

Language mix: TypeScript/JavaScript dominant; Python via zulip, pydantic, drf, ebg; Swift present in ortomate-studio (unsupported, see §5).

All repos are now complete except n8n, whose whole-repo scan never finished (§3.26); its numbers come from four subtree scans. PostHog completed on the second attempt after 3 h 42 m and contributed 38,063 findings (more than four times the nine-repo judged corpus) but arrived after judging closed, so it appears in §3 and §5 and never in an act rate. Sixteen repos attempted, fifteen scanned, ~58,000 files.

---

2. Per-detector verdict

Combined across both judges, detectors with n ≥ 4.

detector	n	act	act rate	0.14+
config_drift	31	5	16.1%	yes
swallowed_error	171	21	12.3%	yes
direct_date	9	1	11.1%
large_file	39	2	5.1%
large_function	197	6	3.0%	yes
boolean_naming_drift	48	0	0%	yes
duplicated_policy	46	0	0%	yes
weak_test_signal	42	0	0%	yes
deep_import	22	0	0%	yes
contract_drift	22	0	0%	yes
sync_io_in_hotpath	16	0	0%	yes
unsafe_retry	13	0	0%	yes
return_shape_roulette	6	0	0%
magic_domain_literal_scatter	6	0	0%
high_fan_in_fan_out	4	0	0%
agent_permission_sprawl	4	0	0%	yes
logic_in_comments	4	0	0%

Nothing clears 17%. The best-performing detectors are among the oldest. Of the ten 0.16 detectors, the two that fired at meaningful volume (swallowed_error, config_drift) are the corpus’s best performers, and the rest scored zero or never fired.

Detectors that fired ZERO times across the whole corpus

On the 9 judged repos (7169 files), 16 of ~64 candidate detectors never produced a finding. Adding airflow and mlflow (15,841 more files, both Python + TS) reduces that list to 11:

timezone_unsafe_parse, permission_ia_drift, copy_ia_drift, action_label_drift, command_drift_docs_code_drift, finder_duplicate_filename, layer_violation (config-gated, expected), plus mixed_utc_local_methods (see §3.21).

Correction: the ten 0.16 detectors have no zeros. On cal.com and n8n (the two large TS services) all ten fire, mock_saturation included (33 and 11). mock_saturation, missing_agent_context and negative_flag_maze were only zero because the earlier corpus lacked the repo shapes they need. That is a meaningfully better result for the 0.16 slate than the first nine repos suggested, and it is a caution about reading a zero as a verdict on a detector rather than on the corpus.

The 0.15.0 cross-language pack: 32 findings across ~58,000 files.

repo	route	type	alias
posthog	1	22	0
airflow	1	5	0
mlflow	1	1	1
everything else	0	0	0

The two halves of this pack deserve opposite verdicts.

cross_language_type_drift is the release’s success story. PostHog: 870 Python closed sets × 1,622 TS unions → 22 real divergences, the best being IntegrationKind drifting in both directions between posthog/schema_enums.py and a generated TS client. Also QuotaResource (Python ↔ Node billing), MetricAggregation, RunStatus, LLMProvider. Caveat: ~11 of the 22 are the same stale codegen reported repeatedly. This detector works and should be kept.

cross_language_route_drift is confidently wrong, not silent: a worse failure than the zero it produced elsewhere. On PostHog it emits one high-severity finding: “41 frontend calls target a path that no backend route declares.” The 28 backend routes it extracted come entirely from two sidecar services (services/llm-gateway, FastAPI, and services/stripe-mock). Zero came from PostHog’s own API, because routes.ts matches only decorator routing while PostHog is Django/DRF: 206 router.register(...), 183 path()/re_path(), 205 @action(...). The backend.length === 0 guard that would have suppressed the finding was defeated by those 28 sidecar routes. On the frontend side, PostHog’s ~2,000 real calls are invisible because they are written api.get('api/…') without a leading slash, rejected by if (!url.startsWith("/")).

So the finding compares PostHog’s test suite against a Stripe mock’s routes, at high severity. It is 100% false. Same shape on airflow, whose route-drift hit is anchored on MSW mock handlers that FIXTURE_RE itself classifies as fixtures.

cross_language_concept_alias_drift cannot fire at scale: PostHog uses every alias in the tenant group (team/workspace/organization/ account/tenant/company) on both sides. The clean vocabulary split it requires is essentially unsatisfiable in a mature product.

---

3. Verified defects

Every item here was reproduced by the orchestrator, not accepted from a subagent report.

3.1 BLOCKER: source under a dot-directory is silently skipped and reported clean

packages/core/src/discovery/files.ts:25 hardcodes dot: false. There is no config or CLI override anywhere in discovery/.

Minimal reproduction, identical 8-line Python file:

as visible.py             → "2 findings across 1 file"
as .agents/skills/bad.py  → "No crimes detected. Suspiciously clean."   EXIT 0

In the wild: ebg keeps all 62 of its .py files under .agents/. coverage.files_by_language had no py key, while packs_loaded still advertised language-py. Renaming .agents → agents moved the scan from 759 to 967 findings and surfaced a real cross_language_concept_alias_drift that the true scan reported as zero.

.github/, .claude/, .config/, .server/ are all common. This is the worst failure mode a scanner can have: a confident all-clear, a cheerful message, and a zero exit code on code it never opened.

3.2 BLOCKER: crimes context ids do not resolve in crimes explain

$ crimes context src/MetricsGrid.tsx   → id=crime_00001
$ crimes explain crime_00001           → file: src/server/migrations.ts

context numbers its findings locally from 1; explain resolves against the global scan ordering. They agree only when the file happens to rank first. context <file> → explain <id> is the documented pre-edit agent workflow and the core of the agent-native positioning. An agent following it receives a detailed, confident rationale for a different finding in a different file, exit 0, no warning.

3.3 BLOCKER: cross_language_route_drift can only ever see a Python backend

packages/core/src/detectors/cross/route-drift.ts:124 skips any file whose pack !== "language-py". A TypeScript backend is invisible in principle, not by calibration.

ebg has 22 app/api/**/route.ts handlers exporting 23 verbs against 9 client fetch("/api/…") call sites: a textbook drift surface the detector cannot flag.

It fired once in the whole corpus: on airflow, a Python backend. Zero on zulip, also Django + TS. So the detector works, narrowly, on the one shape it permits, and structurally excludes the most common polyglot shape in the JS ecosystem: a TS/Next.js backend with a TS client.

coverage.by_package is additionally gated on MIN_PACKAGES = 2 manifest dirs. It is present on airflow and hono, and absent on ebg: the repo shape the feature’s own motivating example describes (“70% TypeScript with one Python service”), because the Python subtree has no pyproject.toml.

3.4 BLOCKER: the 0.17.0 fingerprint fix did not fix the bug class

v0.17.0 broke the wire format to fix fingerprint collisions, reporting “Self-scan collisions: 3 → 0” and “the bug is in the fingerprint function’s inputs, so it gets one general answer.”

The general mechanism shipped. It was wired into 2 detector files (exact-duplicate-block.ts, magic-domain-literal-scatter.ts): the instances the self-scan surfaced. The self-scan was a biased sample: this repo has few test files and little commented-out code.

Measured on real repos:

repo	findings	unique fingerprints	lost to collisions
hono	377	288	89 (24%)
zulip	3759	3433	326
buildfest	1556	1362	194 (12%)
pydantic	578	552	26

Worst groups: commented_out_code ×12 (empty symbol), duplicate_component_shape ×10, large_function ×9 (describe callback), weak_test_signal ×8 (empty symbol).

These are exactly the consequences 0.17.0 named as justifying a wire break. crimes ignore on one commented_out_code finding silently suppresses up to 12. crimes diff is blind to 24% of hono’s findings. Empty symbol is the tell.

3.5 BLOCKER: crime_NNNNN ids are not stable across runs

Two back-to-back scans of hono at the same SHA produced an identical finding multiset in a different order. Ids are positional, so five ids changed identity: run A crime_00024 = large_file on secure-headers.ts; run B crime_00024 = hardcoded_localhost on a benchmark. Suspected cause is the clock-derived recency score drifting 0.71 → 0.70 and flipping a sort tie.

0.17.0 fixed evidence-string determinism. Finding identity ordering is still clock-dependent, and crimes explain <id> does not survive a re-run.

3.6 BLOCKER: agent_risk has collapsed into size

15 of the top 20 findings by agent_risk on ebg are large_function / large_file. Nothing from the 0.16 slate appears, despite parallel_destination firing 52× and orphaned_destination 39× in that repo.

CLAUDE.md states agent_risk “is the differentiator and must not be collapsed into severity”. Observably it is a length ranking. This is the score the entire product thesis rests on.

3.7 BLOCKER: .gitignore is not respected

CLAUDE.md § Stack decisions states: “File walking: fast-glob + ignore (respect .gitignore).” It does not.

Reproduction from scratch:

/tmp/gitig3/.gitignore              → "generated/"
git check-ignore generated/big.ts   → generated/big.ts   (ignored)
crimes scan .                       → "2 findings across 1 file · 1 high"
                                    → generated/big.ts: God Function, God File

The gitignored file is the only file in the report.

In the wild, on buildfest: migrate-output/ is line 8 of .gitignore and a verbatim copy of apps/airquote. It produced 100 findings directly and 229 more citing it (323 of 1556, 20.8% of the entire report) and 136 of 158 (86%) exact_duplicate_block findings. It also appears in coverage.by_package.

Duplication detectors are maximally vulnerable to build output, and build output is exactly what .gitignore names. This stayed invisible because the hardcoded default excludes cover node_modules, dist and .next (the common cases) so only project-specific ignore rules expose it.

Together with §3.1, crimes both scans what it shouldn’t and skips what it should, and reports neither.

3.8 BLOCKER: large_file counts total lines + 1 and calls it “non-empty”

file	crimes claims	real total	real non-blank
pydantic/fields.py	1914	1913	1654
pydantic/types.py	3336	3335	2647
pydantic/networks.py	1371	1370	1084

Exactly total + 1 every time, labelled “non-empty lines”: overstating the true non-empty count by 16–26%.

It manufactures findings. Of 109 large_file findings on pydantic, 11 (10%) would not fire at all on the true count: _migration.py (claimed 318, real 298, threshold 300), _dataclasses.py (316/253), v1/env_settings.py (351/290), five pydantic-core/*.rs, and two markdown files, so 0.17.0’s docs shape inherits the bug (docs/migration.md: claimed 1168, real 889, threshold 1000).

large_file is the second-highest-volume detector in the corpus (779 findings). The lines field and every “N× the threshold” ratio are wrong by the same amount.

3.9 BLOCKER: scan on a nonexistent path exits 0 and reports clean

$ crimes scan /tmp/this-path-does-not-exist-xyz
CRIME SCENE REPORT
repo: this-path-does-not-exist-xyz  ·  0 findings
No crimes detected. Suspiciously clean.
EXIT=0

The JSON form is equally well-formed: valid schema_version, empty findings, files_total: 0, exit 0. A path typo in CI is a permanently green gate. verdict and explain both correctly exit 2 on bad input; scan is the outlier.

3.10 BLOCKER: test-scope classification is structurally inert on Python

packages/core/src/config.ts:17:

export const DEFAULT_NON_DOMAIN_PATTERNS: string[] = [
  "scripts/**", "examples/**", "fixtures/**", "public/**",
  "**/__tests__/**",
  "**/*.test.{ts,tsx,js,jsx}",
  "**/*.spec.{ts,tsx,js,jsx}",
];

No tests/**, no test_*.py, no *_test.py. Measured: every finding is tier: "domain": pydantic 578/578 including 118 under tests/, drf 104/104, zulip 662 test-file findings all domain.

The sharp part: crimes already has the right classifier. util/test-files.ts:23 matches test_[^/]+\.py$ and .*_test\.(?:py|go)$. tier uses a third, glob-based, JS-only mechanism. classifyScope is imported by exactly 4 detectors, all 0.16.0 JS-side: zero Python and zero universal detectors consult it.

CLAUDE.md presents util/scope-class.ts as “one answer to generated / vendored / migration / fixture / test / config / production”. There are three answers, and the one wired into scoring cannot see a Python test. Test files are 62–64% of both pydantic and drf.

3.11 BLOCKER: silent, nondeterministic finding loss under fd pressure

At ulimit -n ≥ 48 buildfest output is byte-identical to baseline: the 0.17.0 bounded pool is correctly sized and holds. Below ~48 the scan silently reports fewer findings, nondeterministically: three runs at ulimit -n 40 on an unchanged tree gave 1522 / 1535 / 1543. Every one exited 0, empty stderr, an identical coverage block, summary.high pinned at 187, while ~11% of the report vanished.

Root cause is not indexes.ts. ast-hash/function-index.ts:72-79:

try { source = await readFile(abs, "utf8"); }
catch { return hits; }   // EMFILE becomes "this file has no functions"

Same shape at jsx/shape-index.ts:65-66. The bare catch cannot distinguish ENOENT from EMFILE, so each exhaustion silently deletes one file from the duplicate corpus: partial and racy, strictly worse than the all-or-nothing undefined indexes.ts was designed to produce. Margin between correct and silently wrong: 8 descriptors.

3.12 crimes context drops exactly the findings an agent cannot derive

On JobDetail.tsx, scan reports 12 findings and context reports 11. The missing one is duplicated_policy: a cross-file finding, i.e. the single class of thing an agent reading that one file could not work out for itself. context is the product’s best surface and its whole purpose is to brief an agent before an edit.

Also: on a pnpm workspace, context app/src/middleware.ts prints file: src/middleware.ts: a path that does not exist, and is plausible because the repo has a real top-level src/. It analysed the right file and displayed an unopenable path.

3.13 BLOCKER: dependency_provenance_gap checks Python imports against npm manifests

zulip, high severity, on package.json:

450 external package(s) are imported but declared in no applicable package.json. They resolve today through hoisting; a clean install can fail.

_typeshed — imported at zerver/lib/test_console_output.py abc — imported at corporate/lib/stripe.py aioapns — imported at zerver/lib/push_notifications.py

abc and _typeshed are the Python standard library. aioapns is declared, in pyproject.toml. Every cited import is from a .py file; the manifests searched are package.json and starlight_help/package.json.

The implied fix is to add the Python stdlib to package.json. Note the same detector’s second finding on the same repo (@typescript/native declared but absent from pnpm-lock.yaml) is plausibly correct. The JS path works; the language dispatch is the defect.

3.14 BLOCKER: blast_radius_importers prints a component size as a per-file count

Verified from zulip’s scan JSON: 3,759 findings carry blast_radius_importers and there are only 37 distinct values. 866 findings share the value 798; 825 share 324. blast_radius is pinned at 1.0 on 1,758 of 3,759 (47%).

In the human report zerver/data_import/slack.py renders as “blast top-quartile (798 importers)”. Actual direct importers: 5. inbox_ui.ts prints 324; actual 4.

Hundreds of unrelated files cannot each have exactly 798 importers: this is a connected-component size rendered as direct fan-in. It is a number a maintainer can disprove in five seconds, printed beside findings that are otherwise correct. explain renders it honestly (“50+ transitive importers (cap reached)”); scan and context do not, so the same score is truthful in one renderer and false in two.

3.15 The cross-language diagnosis, on the second ideal input

zulip: route_drift 0, type_drift 0, alias_drift 0. Both halves are blind, and the cause is idiom coverage, not calibration:

• routeFromDecorator() is the only route source and reads decorators only. Zulip declares 387 URLs as Django URLconf function calls: rest_path("realm/domains", GET=…, POST=…). Repo-wide @app.get-shaped decorators: 0. So the detector early-returns on an empty backend set and never reaches its comparison. Secondarily, Django routes are relative ("realm") and the extractor rejects anything not starting with /.
• The frontend half fails independently. Zulip makes 140 calls shaped channel.post({url: "/json/typing", data, …}). The receiver channel fails REQUEST_RECEIVERS = /(^|\.)(axios|http|client|api|request)$/, and the URL sits in an object-literal first argument, so literalText() returns undefined. swallowed_error.js parses that exact call fine and quotes it in the human report: two collectors in the same parser disagree about what an HTTP call is.
• type_drift did run and honestly found no name matches, but its Python pool is junk: the allValued fallback accepts any class whose members are all string constants, so 14 of 19 “closed sets” are things like Integration (image-path templates). Zulip’s 29 real enums are IntEnum/Enum with integer values, which the extractor never records.

And none of this is visible from the output: "cross-language" is never a member of packs_loaded: that array is ["universal", ...registeredPacks()]. No JSON field distinguishes “ran and found nothing” from “never ran.”

3.16 BLOCKER: no size or line-length guard before parsing (135× slowdown)

ortomate-studio: scan --all --format json takes 675 seconds. The same repo with 92 vendor bundles removed takes 5 seconds. One 1.9 MB bundle alone accounts for 251 seconds. The repo’s longest line is 1,621,186 characters. diff HEAD~5...HEAD never completed.

Root cause, verified at packages/core/src/config.ts:216-217: the default excludes are **/*.min.js and **/*.generated.*: filename patterns only. Modern bundlers emit name-HASH.js (mermaid.core-DKFhFOEB.js), which matches neither. There is no size guard, no line-length guard, and no --exclude or --max-file-size flag on scan: so under --no-init there is no workaround.

The compounding harm is worse than the latency. scope-class.ts classified these bundles as production domain code, so 444 of 851 findings (52.2%) land inside a vendored reference/ directory, and 15 of the top 20 by agent_risk are minified third-party bundles. All five entries in the default “Top files by risk” are somebody else’s code, with findings reading Unprefixed Boolean 'So' @L1. The report’s closing call to action is:

→ Start with crimes context chrome-extension/reference/claude-1.0.77/assets/mcpPermissions-CJK8I7C7.js

3.17 BLOCKER: absent signals are scored as zeros and rendered as good news

Across all 175 Swift findings in ortomate-studio, blast_radius = 0, blast_radius_importers = 0 and test_gap = 0: constant, not distributed, because no pack claims Swift. The schema has no “unknown”, so absence becomes the best possible score.

Sharpest instance: crimes context ProjectsStore.swift (4,949 lines, 97 commits in 90 days, no test file) prints test gap: bottom-quartile, the identical phrase a well-covered TS file gets. Swift test discovery looks for .test/.spec/__tests__ and misses Tests/OrtomateStudioTests/*Tests.swift (20+ files, standard SwiftPM).

In the command designed to brief an agent before it edits, the tool reports the opposite of the truth. Same family as test_gap: 0.5 narrated as a causal sentence and churn: 0.00 rendered as “churn low”.

3.18 The honesty gap is one line of output wide

This one is mostly good news and is the cheapest high-value fix in the round.

On ortomate-studio (429 Swift files, 85% of source LOC) the JSON is completely honest: all 429 files appear in files_total, and universal_only_by_extension names ".swift": 429. --explain-coverage prints an unusually good sentence:

These files were not claimed by any loaded language pack. They received universal-pack checks only (file size, raster assets, duplicate filenames, hardcoded localhost/local paths, docs link checking).

That flag is off by default and printed last, after the report it should be qualifying. In the 174-line default report the string “swift” appears zero times. The 175 Swift findings come from 4 detector types; ~45 detectors ran on 16,012 lines of TS/JS and on none of the 91,778 lines of Swift.

Promoting one header line (429 of 619 files: universal checks only (.swift)) closes the gap entirely.

3.20 BLOCKER: 41% of airflow’s findings are the Apache license header

commented_out_code fired 8,019 times on airflow: 45% of all 17,801 findings. Of those, 7,320 have a line range of exactly (1,16) or (1,17): the ASF licence header on every source file. That is 41.1% of the entire report, verified from the scan JSON.

The evidence line reads “≥3 lines contain code-shaped tokens (def/fn/function/class/let/var/const/etc.)”: the header contains none of those. The likeliest triggers are the http:// URL and the quoted "License" / "AS IS" string literals.

mlflow, which carries no per-file headers, produced 151. Every Apache-licensed repository will see this, and Apache-licensed repositories are a large fraction of serious open-source Python and Java. This is the single largest noise source found in the round.

Related: on zulip the same detector flags long English prose comments (a 15-line explanation of a security invariant) because the words “class” and “function” appear, with the summary “agents may revive incorrect snippets”. There is nothing to revive.

3.21 mixed_utc_local_methods cannot fire on modern Python: diagnosed

Zero findings across 10,304 real Python files, including Apache Airflow, which has 775 utcnow() call sites, 354 datetime.now( call sites, and 21 files containing both, 9 of them non-test production modules.

Isolated with a synthetic probe: the detector fires correctly on the bare form (datetime.utcnow() + datetime.now() in one module) and does not fire when the UTC read goes through a wrapper (timezone.utcnow()). direct_date also skips the wrapper call, so the parser’s dateCalls never sees <module>.utcnow() at any layer.

The missed real-world positive: airflow’s kubernetes_executor.py has naive datetime.now() at lines 464 and 630 and timezone.utcnow() at 1098, same module. direct_date did fire on that file at [464, 630], proving the parser saw the local reads.

Every large Python codebase wraps clock access, and bare datetime.utcnow() is deprecated in 3.12, so the detector is structurally unable to fire on the Python it targets.

3.22 Scan cost tracks the JS pack, not repo size

mlflow is 30% smaller than airflow and has a third of its Python, yet scans 2.3× slower (402s vs 173s). The only axis where it is bigger is JavaScript: files_by_language shows js: 3513 vs airflow’s 1089.

Consequences: verdict and diff run two scans each and are structurally >13 minutes on mlflow: both were killed at 10 minutes. context, which the crimes skill instructs agents to run before every edit, takes 97–112 seconds per file on mlflow.

3.23 More silent-degradation instances

• UTF-16 and NUL-byte source files produce zero findings, are counted in coverage.files_by_language.py, exit 0, empty stderr. Latin-1 decodes fine via replacement. There is no files_failed counter. (Synthetic probe; byte-identical Python bodies.)
• 42 .ipynb notebooks in mlflow produced zero findings and appear in neither files_by_language nor universal_only_by_extension: silently outside coverage entirely.
• Generated code is not excluded. GENERATED_RE does not match _pb2.py, so mlflow’s mlflow/protos/ drew 28 findings including service_pb2.py at 5,696 lines. A synthetic 60,002-line file whose first line is # @generated by protoc. DO NOT EDIT. still drew large_file, so looksGeneratedSource is not being called either. Worse: airflow’s generated OpenAPI client drew 125 findings and 6 of the top 20 by agent_risk (including rank 3 overall), and two of those paths (services.gen.ts, schemas.gen.ts) do match GENERATED_RE and are isNeverReportable, yet were reported anyway.

3.24 sync_io_in_hotpath aggregates across a file but wears one symbol’s label

mlflow’s worst case is labelled _hard_delete_experiment() with lines spanning 900–9073: 81% of a 10,093-line file, covering 15 call sites in different methods. Twelve of the fifteen are SQLAlchemy session.delete() / session.get(), classified as family network.

This has a direct consequence for this round: any consumer using lines to build a ±30-line excerpt gets a meaningless window: including the blind-judging cards used here. See §9.

3.26 BLOCKER: n8n cannot be scanned at all, and cost is superlinear

n8n: 26,079 tracked files, 20,050 .ts/.vue/.js. Two attempts:

• killed at exactly 60 minutes, still CPU-hot, 0 bytes of output
• fully detached so nothing could reap it: 2 h 51 min and still running when the round ended. Still 0 bytes.

Output is buffered to the very end, so an interrupted scan yields nothing at all. There is no --progress, no --timeout, no --concurrency, no --exclude. From outside, a slow scan and a hung one are indistinguishable. cal.com completed, at 7 m 38 s.

Cost is superlinear, measured on both repos across three scopes each:

repo	small	large	ratio
n8n	255 files → 6.7 s	2,977 → 234 s	11.7× files, 35× time (≈O(n^1.45))
cal.com	428 files → 6.0 s	5,051 → 458 s	11.8× files, 76× time (≈O(n^1.75))

Per-file cost at small scope is fine (24–26 ms). Something cross-file is quadratic-ish. docs/ claims the risk indexes use “bucketed matching, never global pairwise”: the curve says at least one is not bucketed tightly enough. 20k files is ordinary for the target segment.

3.27 BLOCKER: .vue files are silently skipped and coverage does not say so

1,226 .vue files repo-wide in n8n (1,037 in editor-ui alone) produced zero findings, and .vue appears in no coverage field: not files_by_language, not universal_only_by_extension. Not even the language-agnostic universal detectors ran on them.

This is precisely the failure the protocol’s robustness section exists to catch, and coverage does not catch it. Same family as the notebooks, the .hbs/.html templates, the .sql files, and the images that produced findings while appearing in no coverage field. A files_skipped_by_extension counter would close all of them.

3.28 dependency_provenance_gap’s undeclared-imports arm is broken generally

§3.13 showed it checking Python imports against npm manifests. cal.com shows the same arm failing on pure TypeScript. One high, confidence: 0.74 finding: “148 external package(s) are imported but declared in no applicable package.json … a clean install can fail.” The 8 itemised examples (the only ones a reader ever sees, the rest is “+140 more”) are:

_utils, ., .., @components/AddToHomescreen, @components/apps,
@components/auth, @components/booking, @components/dialog

Two are relative specifiers. Five are a TypeScript path alias declared in apps/web/tsconfig.json: cal.com has no root tsconfig.json, which is normal for a Next.js monorepo, and alias resolution only reads a root one. The finding’s own evidence promises “node built-ins, relative paths, path aliases … are excluded”. All three exclusions failed, on a high-severity finding written to alarm.

The rest of the detector is genuinely good and worth protecting. coverage.by_package on cal.com is 118 entries against a ground truth of 118; the lockfile arm reports “114 workspace manifests compared; 5 outside the workspace skipped” and the 5 are exactly packages/app-store/templates/*. The 0.17.0 !-negation glob work is doing its job. It is one arm that is broken, not the detector.

3.29 pass_through_abstraction: third and fourth independent confirmation

§4 recorded it fabricating a 5-layer chain in zulip from get method names. Both cal.com and n8n reproduce it, and worse: confidence rises with the number of unrelated files pulled in. cal.com’s most confident hit is 0.92, joining three unrelated repositories on the method name delete; n8n’s is 0.98, joining four unrelated registries on has.

The single-file arm looks sound and is a different code path. It is the chain arm that should come out.

3.30 The default view hides every repo-level finding

scan groups by file, so any finding anchored on a repo-level artefact never appears. On cal.com the scariest finding in the entire scan (dependency_provenance_gap, high, “148 external packages … a clean install can fail”) is anchored on package.json and is not in the default report at all. Nor are the 33 mock_saturation, the 55 config_drift, or agent_permission_sprawl. In n8n packages/cli, seven of the ten 0.16 detectors appear zero times in the default view despite firing in the JSON.

Meanwhile four of cal.com’s five “Top files by risk” are test scaffolding (playwright/fixtures/users.ts, packages/testing/.../bookingScenario.ts), and the closing call to action points a first-time user at a test helper as “the most risk in this scan”. The report has a tests bucket, but it matches a literal tests/ prefix and catches none of playwright/, *.e2e-spec.ts, or packages/testing/.

So the ranking answers “which file has the most findings”, not “what should I worry about”, and the differentiated 0.16 findings, which are the product’s actual wedge, are the ones it hides.

3.32 BLOCKER: crimes init writes a JS-only config and silently amputates Python

packages/cli/src/init-detect.ts:85:

const include = ["**/*.{ts,tsx,js,jsx,mjs,cjs}"];

Hardcoded. detectRepoShape() recognises only pnpm / turbo / next / vite. So the documented first-run setup (and crimes init --agents, which the docs tell every agent user to run once) writes a config that excludes every .py file in the repo.

Measured on PostHog’s products/metrics: 100 files → 37, Python 48 → 0, findings 75 → 33, cross_language 1 → 0. Repo-wide the generated config costs 23,273 of 38,063 findings (61%).

And packs_loaded still reports language-py in both cases. The init help text documents the eight Python detectors that the config it just wrote has disabled.

3.33 The quadratic cost, root-caused

§3.26 measured superlinear cost. PostHog’s CPU profile names it: 36% of samples sit in the anonymous comparators of exact-duplicate-block.ts and near-duplicate-block.ts, which materialise and localeCompare-sort the entire global function-hash index once per file, then discard everything not anchored on that file. That is O(files × functions × log functions).

Python scales fine (4,759 .py files in 55 s). JS is 22.6× worse per file. PostHog’s full scan: 3 h 42 m, 1.70 GB max RSS, 2.36 GB peak footprint, 38,063 findings, 9,177 high.

This is a contained fix (sort the index once, or bucket by hash before sorting) and it is the difference between crimes being usable and unusable on a large monorepo.

3.34 More correctness problems from the largest repo

• magic_domain_literal_scatter is non-monotonic. 194 findings on frontend/, 3 on products/metrics, and 0 on the whole repo that contains both. A scan of a parent directory should not lose findings a scan of its child produced.
• baseline save then baseline check disagrees on an untouched tree: save 33, check reports 30 unchanged. 2,561 of 38,063 findings share a fingerprint with another; one ignore can silence 19.
• Python test conventions are absent from nonDomain, confirmed at scale: 4,301 findings in pytest files are tier: domain, while all 2,883 JS test findings are correctly nonDomain. This is §3.10 with a much larger sample and a clean control inside one repo.
• triage --apply silently writes a known entry into feedback.jsonl, and requires an undocumented strict document schema (a bare array is rejected). It is the only non-interactive path, i.e. the only agent/CI path.
• Config is resolved from the scan root, not cwd. crimes init at the repo root then crimes scan subdir/ silently ignores your config. crimes diff ignores it too (72 vs 33 findings, unexplained).
• --show-suppressed never annotates anything, and still prints “run with —show-suppressed to see” while you are running with it. Same for --show-triaged.
• The #1 recommendation is a generated file, again: → Start with crimes context frontend/src/queries/validators.js: line 1 is // AUTO-GENERATED FILE - DO NOT EDIT DIRECTLY, 13,728 lines of AJV output, 52 findings, 36 high. The first 25 lines of the entire report are God Function · validate12() … validate94().

3.35 SYSTEMIC: detectors with OR-triggers print evidence for the condition that did not trip

A detector fires on A || B but builds its summary and leading evidence from A unconditionally. When only B tripped, the finding presents a measurement that passed as if it were the reason.

• py/large-function.ts:75-108: fires on tooLong || tooDeep, always says “is N lines … At this size an agent must read the whole body”.
• todo-density.ts:40: if (matches.length < 3 && density < threshold) return [], then prints “4.2 markers per 1k LOC (threshold 10)”.

Real output: pydantic get_strict (14 lines, threshold 50) charged as a God Function. deep_update: 9 lines. drf: 12 of 50 sampled cards. zulip: 26, 19, 23 lines against a 50-line threshold.

All three blind judges independently used the word “contradicted”. The underlying signals (deep nesting, TODO count) are legitimate; the message destroys them. This is the round’s cheapest fix and its most expensive trust cost.

---

4. False positives in full

The detectors below are not miscalibrated at the margin: they assert things the code in their own evidence window refutes.

pass_through_abstraction fabricates call chains from name collisions. zulip: reported get_services “forwards through 5 layers across 5 files”. The body is return services.get(bot_id): one line. The chain was assembled from unrelated get methods in markdown_config, message_list, sub_store. 19 findings, all one repo.

weak_test_signal does not follow assertion helpers. 42 judged, 0 act. drf-0033: “11 of 11 test functions assert nothing”: every test routes through _resolve_urlpatterns, which the card itself prints asserting on three attributes. drf-0037: 10 tests call _assert_datetime_rendering, shown ending in self.assertInHTML. It counts self.assert* but not self.fail, not model_dump_json(warnings='error'), and treats @pytest.mark.benchmark functions as assertion-free when measuring is their entire purpose. Extract-a-shared-assertion-helper is among the most common test idioms there is; the detector reads it as the absence of testing, and is most wrong on exactly the well-factored suites it should be quietest about. 619 findings corpus-wide.

boolean_naming_drift flags names that cannot be renamed. 48 judged, 0 act. Four findings on atomic = False inside class Migration(migrations.Migration): a Django-defined attribute. 18 names “bound to a boolean expression with nothing saying they hold true/false” that are Pydantic fields explicitly declared name: bool. Names its own convention clause exempts (“or an idiomatic bare name”): nullable, plural, frozen, from_attributes, lock_acquired. On p-limit it proposes renaming rejectOnClear: a public API option, a semver-major change: at effort: "quick".

sync_io_in_hotpath has no working notion of “hotpath”. 16 judged, 0 act. Fires on a if __name__ == "__main__" setup CLI, a Django management command, a release/push.py script, an @cache-decorated once-per-process function, and two calls inside if settings.DEVELOPMENT: next to a comment saying nginx serves the file in production. On ebg it is also a file-level finding wearing a function’s name: 43 findings across exactly 43 files, with symbol and lines spanning ranges the named function does not occupy.

swallowed_error does not read the comment that explains the catch. Best-performing 0.16 detector at 12.3%, but 20 of hono’s 43 cards were this detector and nearly every one had an explanatory comment or a visible fallback. It fires where the next line is throw err (hono-0011); where the catch calls three recovery functions; and on p-limit’s index.js:44, three lines below a comment reading “We catch errors here to prevent unhandled rejections, but the original promise rejection is preserved for the caller.” Its BEST_EFFORT_FUNCTION_RE recognised deliberateness on exactly one hono card while missing an identical comment two functions above.

unbounded_async_fanout on p-limit: a library whose entire purpose is bounding concurrency. Its evidence simultaneously states “collection source: an awaited call” and “produced by: Array.from”, about Array.from({length: 100}, …), then concludes “N is a property of the data, not of this code.”

Vendored and framework-owned code is treated as domain code. drf: three findings on vendored minified google-code-prettify. pydantic: 10 cards target pydantic/v1/, the frozen vendored tree. Django’s own mandated management/commands/* layout flagged as reaching into internals. util/scope-class.ts is documented as the single answer here and is not catching vendored trees, framework layouts, or frozen legacy subtrees. This inflates every downstream count.

---

5. Actionability, separately from correctness

The act-rate table is the actionability measure and it is the report’s core finding. Three observations the precision framing would miss:

Volume is the product. 8134 findings across the 9 judged repos; zulip alone 3759 with 502 high. Airflow, scanned after judging closed, returned 17,801 findings with 929 high on 9149 files: of which commented_out_code alone is 8019 (45%) and deep_import 3453. Even at a generous 15% act rate, an airflow maintainer would sift ~15,000 findings to reach ~2,600 worth considering. “Signal over exhaustiveness” is a stated constraint; at this volume it does not hold, and the two detectors driving it on large Python repos were not even in the judged sample.

Default-view suppression is a file cap, not a finding budget. scan.topFiles: 5. On hono that compresses 377 → 35 (9% shown). On p-limit it shows 9 of 10 (90%). The compression is an accident of repo size rather than a designed budget: the small clean repo, where a first-time user calibrates their trust, gets no compression at all.

The loudest outputs are the least reliable. Judges consistently rated crimes context the most useful command and scan’s headline and hotspots the least. On hono, hotspots ranks package.json #1 at 72% risk (14 points clear of anything real) because it churned 29× in 90 days on version bumps. That ranking is backwards from what earns trust on first contact.

--all is a byte-for-byte no-op in --format json: verified by identical MD5 on two repos, while the human output advertises “Run with --all to see them.”

---

6. Robustness and performance

Python parsing: PASS. This is the round’s clearest good news. On ebg, 51 of 62 .py files produced findings, with positive evidence per construct: match statements bounded correctly and counted case arms as nesting; walrus across 5 files; PEP 604 across 8. PEP 695 generics, type X =, and except* were probed synthetically and parse with correct symbols and ranges. No parse failures, no crashes, no grammar errors anywhere in the corpus.

The zero case: PASS. On every single-language TS repo, all three cross_language_* detectors and all Python detectors produced exactly zero. No false positives from either pack. But coverage.packs_loaded lists language-py on repos with no Python: it reports what was loaded, not what ran, and disagrees with files_by_language. Only the latter is informative, and the protocol’s own advice to “check packs_loaded” is therefore wrong.

Determinism: mixed. Finding sets were reproducible (ebg: two runs byte-identical). Finding order and ids were not (§3.5).

Performance. Scans are acceptable (zulip: 3588 files in 81s). The git-touching commands are not: on ebg, a 6s full JSON scan sits beside hotspots at 27s, verdict at 53s, and diff at 62s: the latter to report “New: 2” without saying which two. verdict is pitched as a CI/pre-commit gate. Three commands run 3–4× wall-clock over CPU.

No EMFILE, no OOM on any completed scan. The 0.17.0 bounded-pool fix held on everything measured so far; the four largest repos were still running at time of writing.

A killed scan leaves a 0-byte artefact indistinguishable from a successful empty one. Several artefacts were 0 bytes from SIGTERM (exit 143), not from clean results.

---

7. Ranked recommendations

Not applied. Each is labelled with its cost per CLAUDE.md and evals/README.md.

Fix before anything else: correctness, not calibration

0. Fix the four silent-wrong-answer bugs as one workstream. They are the same failure: crimes knows something is off and says nothing.
   • .gitignore ignored (§3.7): scans what it shouldn’t
   • dot: false (§3.1): skips what it should scan
   • nonexistent path → exit 0 “clean” (§3.9)
   • EMFILE swallowed as “no functions” (§3.11) Add a single coverage.warnings[] and make every one of them populate it. No output should ever say “Suspiciously clean” when anything was skipped, unreadable, or absent. Cost: schema addition (minor) + four small fixes.
1. Scan dot-directories, or say you didn’t. dot: false → configurable, defaulting to scanning them minus a deny-list. At minimum, never print “Suspiciously clean” when files were excluded. Cost: detector-adjacent product change: patch bump + eval re-run. 1b. Respect .gitignore: the behaviour CLAUDE.md already claims as a locked stack decision. Grep confirms no gitignore parsing exists anywhere in discovery/. Expect a large finding drop: buildfest loses 20.8%, mysidekick-cc 19.1%. hotspots already drops untracked files for free via its churn ranking: scan could borrow that same tracked-in-git check and fix most of this plus the vendored-bundle problem at once. Cost: patch bump + eval re-run. 1e. Add a size / line-length guard before parsing, and a --exclude / --max-file-size escape hatch (§3.16). The default excludes are filename-only and miss every modern content-hashed bundle. This is the difference between a 5-second and an 11-minute scan, and between 52% vendored noise and none. Cost: config default + a scope-class fix. 1f. Give the schema an “unknown” so an unmeasured test_gap or blast_radius cannot render as the best possible score (§3.17). Cost: schema change: minor. 1c. Fix the large_file line count (§3.8). It is counting total + 1 and calling it non-empty. ~10% of large_file findings are phantom. Cost: patch bump + eval re-run. 1d. Wire Python and universal detectors into classifyScope and add Python test globs to DEFAULT_NON_DOMAIN_PATTERNS (§3.10). Until then no Python finding is ever discounted for being in a test. Cost: patch bump + eval re-run; expect a large scoring shift.
2. Make context ids resolvable by explain. Either emit global ids from context or have explain accept a file-scoped id. Cost: CLI fix, no schema change.
3. Give every multi-finding-per-file detector a discriminator. Empty symbol is the tell: weak_test_signal, commented_out_code, duplicate_component_shape, large_function. Assert fingerprint uniqueness in the scan pipeline so it cannot regress. Cost: fingerprint change for those types: minor + migration note, same shape as 0.17.0.
4. Make finding ids deterministic. Remove clock-derived recency from the sort key, or make ids content-derived rather than positional. Cost: patch bump + eval re-run.
5. Branch OR-trigger summaries on the condition that tripped. Audit every compound-trigger detector. Start with py/large_function and todo_density. Cost: prose-only for most; patch bump + eval re-run.

Then: the detectors that earn their place

6. Sunset or gate the zero-act detectors. See §8.
7. Teach weak_test_signal to follow assertion helpers, count self.fail, and exempt @pytest.mark.benchmark. Until then it should not ship at 619 findings. Cost: detector tuning: patch bump + eval re-run.
8. Give sync_io_in_hotpath a real hotpath test, or rename the charge to what it actually detects (blocking I/O anywhere). Cost: detector tuning, or prose-only if renamed.
9. Widen scope-class to vendored trees, framework-mandated layouts, and frozen legacy subtrees. Cost: patch bump + eval re-run.
10. Exempt framework-owned and public-API identifiers from boolean_naming_drift, and stop proposing semver-major renames at effort: "quick". Cost: detector tuning.

Then: honesty of the report

11. packs_loaded → report what ran, with per-pack file counts. Cost: schema addition: minor.
12. Make the default view a finding budget, not a file cap. Cost: config default.
13. Fix --all in JSON mode: either honour it or warn that JSON is always complete. Cost: CLI fix.
14. Promote large_file shape to a schema field. It exists only as an English sentence in evidence; tier is a different axis. Cost: schema addition: minor.
15. Add json/yaml/txt/rst/adoc to DEFAULT_SOURCE_INCLUDES or stop documenting policies for them: 5 of the 7 extensions the 0.17.0 docs shape claims are unreachable at default config, as is the deliberate data-format exclusion. Cost: config default.
16. Split score-derivation out of evidence into score_rationale. Cost: schema change: minor.
17. explain must handle every type the scanner emits: it exits 2 on oversized_raster. Make the copy-paste crimes ignore block shell-safe. Cost: CLI fix.
18. hotspots: exclude manifest churn, scan the same file universe as scan, and say “nothing is hot” when nothing is. Cost: detector tuning.
19. Speed up or de-scope verdict / diff: 53–62s to answer “unchanged” is not a pre-commit gate. Cost: engineering.

---

8. Sunset shortlist

Detectors with zero act verdicts from both judges at n ≥ 4, or which never fired at all.

detector	evidence	recommendation
cross_language_route_drift	3 findings in ~58,000 files, and the two examined are 100% false: PostHog’s compares its test suite against a Stripe mock, at high severity	Remove or rewrite. It only matches decorator routing, so it misses Django/DRF and TS backends entirely, and its empty-backend guard is defeated by sidecar services
cross_language_type_drift	28 findings, real divergences: IntegrationKind drifting both ways, QuotaResource, RunStatus	Keep. The 0.15 release’s genuine success
cross_language_concept_alias_drift	1 finding in ~58,000 files; requires a clean vocabulary split that mature products do not have	Default off
mock_saturation	0 findings: 33 on cal.com, 11 on n8n, 100% in test files, one per test case	Keep. The earlier zero was a corpus gap, not a detector fault
parallel_destination	2,819 findings from 134 files: 53% of one n8n package; 0 on cal.com and 0 on n8n packages/cli	Default off. Pairs Vue composables on the token use, the mandatory prefix of every composable. Dormant until it hits a naming convention, then it drowns the report
duplicated_policy	46 judged, 0 act, both judges	Remove or gate. The largest zero in the round
weak_test_signal	42 judged, 0 act; 619 findings corpus-wide	Fix (helper-following) or remove. Highest noise volume
boolean_naming_drift	48 judged, 0 act; 637 corpus-wide	Gate to --all pending an exemption pass
contract_drift	22 judged, 0 act	Gate to --all
deep_import	22 judged, 0 act	Gate to --all
sync_io_in_hotpath	16 judged, 0 act	Fix the hotpath test or rename
unsafe_retry	13 judged, 0 act; fired next to maxAttempts: 3, factor: 2	Fix: it has a correctness bug, not a taste problem
pass_through_abstraction	fabricates chains from name collisions	Remove until fixed
agent_permission_sprawl	4 judged, 0 act	insufficient n: re-measure
missing_agent_context, timezone_unsafe_parse, permission_ia_drift, copy_ia_drift, action_label_drift, command_drift_docs_code_drift, finder_duplicate_filename, negative_flag_maze	0 findings	Justify or remove: a detector that never fires in the wild is a finding about the detector

What to keep and lead with. crimes context was rated the most useful surface by every judge that used it: related files, likely tests, per-file findings, agent guidance. It is the actual wedge. The evidence blocks are genuinely well built: shown arithmetic, line numbers, named fix shapes, nothing bare-asserted. The tool reasons honestly from bad premises. Fix the premises.

---

9. What was deliberately not judged

• Findings outside the sampling rule. The rule was “every finding from a 0.14+ detector, plus the top 20 by agent_risk”, randomly capped to 50 per repo at seed 20260802 where it overflowed. 426 of 8134 findings were judged. The cap was applied mechanically, never by picking.
• Five repos still scanning at time of writing: posthog, mlflow, n8n, cal.com, ortomate-studio. Three others (airflow, choreograph.cc, mysidekick-cc) completed after the judging sample closed and are counted only in the volume and zero-firing analyses, never in an act rate. Airflow materially changed one conclusion (the cross-language pack is not dead, only nearly so) which is a reminder that this corpus is still small enough for one repo to move a headline.
• The write commands (init, baseline, triage, ignore, feedback) were exercised only on disposable clones and are reported in the friction log, not judged for usefulness.
• crimes feedback was not used to record these verdicts. The protocol called for it. The fingerprint collisions in §3.4 mean recording 426 verdicts by fingerprint would have silently merged an estimated 10–24% of them. That is itself a finding: the calibration loop cannot currently record the results of a calibration round.
• No detector was changed during the round, per the protocol.
• A known flaw in this round’s own instrument. The blind-judging cards were built by taking a ±30-line window around each finding’s lines field. §3.24 shows that some detectors report a lines span covering most of a file, so a minority of cards gave the judges a truncated or meaningless window. Those cards were more likely to be judged noise for reasons that are the card builder’s fault rather than the detector’s. This inflates the noise rate by an unknown but probably small amount: the affected detectors (sync_io_in_hotpath, boolean_naming_drift) are ~26 of 426 judged cards. It does not touch the act rates of duplicated_policy, weak_test_signal, large_function or swallowed_error, which carry accurate ranges. Separately, 12% of findings carry no lines field at all (high_fan_in_fan_out, exact_duplicate_block, near_duplicate_block, duplicated_role_status_plan_check, circular_dependency, deep_import), and symbol is undefined on 20 of 34 types, so those cards fell back to the first 60 lines of the file. Any future round should fix the card builder first, or better, crimes should carry a usable range on every finding.
• PostHog never completed and is absent entirely. n8n’s whole-repo scan also never completed (§3.26); its numbers come from four subtree scans, which is not equivalent: coverage.by_package is absent from every subtree scan, so n8n’s workspace handling went untested.
• Judge disagreements (26% of cards) are recorded in ~/crimes-dogfood/disagreements.json and were not adjudicated individually; the two judges are reported separately throughout rather than merged into a single number.

---

10. Assessment

The question was whether crimes is genuinely useful. On this evidence:

Not yet, in its default configuration, on well-maintained code. Five mature repos, 203 judged findings, two worth acting on.

But the failure is not that the ideas are wrong. It is that a cluster of defects is poisoning a large amount of correct work, and they share one shape: crimes knows something went wrong and says nothing.

• a walker that skips hidden directories and calls the repo clean
• a walker that ignores .gitignore and charges you for build output
• a nonexistent path that returns a well-formed clean report, exit 0
• descriptor exhaustion that silently deletes 11% of findings, racily
• a line counter that reports total + 1 as “non-empty”, inventing ~10% of its own findings
• an id scheme that survives neither a re-run nor a second command
• fingerprints that collide on up to a quarter of findings
• summaries that state the opposite of their own evidence
• test-scope classification that cannot see a Python test
• a differentiating score that has quietly become a line count

None of these require rethinking the product, and most are small. What they have in common is more important than any one of them: the tool’s default response to uncertainty is reassurance. For a product whose entire pitch is evidence before judgement, that is the thing to fix first: a single coverage.warnings[] would surface four of them at once.

The product hiding inside this one

The most useful number in the round is not the act rate. On zulip, 81% of the report (3,038 of 3,759 findings) is line counts, naming, commented-out code and TODO density: the bucket CLAUDE.md explicitly says crimes is not trying to compete in. large_function alone is 39% of the report and agent_risk ranks it first: the top 20 by agent_risk is 18 large_function/large_file findings and zero Python, on a repo that is 71% Python.

Strip that bucket and 721 findings remain: about 1.2 per 1,000 lines. That is a defensible number for a repo of zulip’s size, and it is roughly the density a maintainer would tolerate.

So the corrective is not primarily “fix the detectors”. It is:

1. Fix the silent-failure cluster, because nothing else matters if the tool will report a clean bill of health on code it never read.
2. Stop agent_risk ranking by length, which is what surfaces the linter-bucket findings first and buries the differentiated ones.
3. Gate or sunset the zero-act detectors so the remaining ~19% is what a user actually sees.

The wedge (local, agent-native, cross-file risk that a linter cannot see) is real and is present in the data. crimes context finds it. duplicated_policy, parallel_destination and orphaned_destination fire on exactly the shapes ESLint cannot reach. They are currently ranked below a function that is 1.0× its threshold. crimes context, the evidence blocks, the Python parser, the zero-case discipline, and verdict’s refusal to manufacture drama are all genuinely good and worth building on.

The honest summary is that the last four releases shipped detector surface faster than they shipped confidence in it, and the self-scan that stood in for real-world validation was a biased sample in exactly the ways a self-scan is always biased. This round is the correction.