crimes@0.25.0

Six changes to what the scanner reads, what it skips, and how it reports the difference.

All finding counts are measured on real repositories: airflow (9,924 findings), mlflow (6,468), pydantic (487), cal.com (4,633), hono (377). Every figure carried in from the backlog was re-derived before it was built on; four did not survive that check and are listed at the end.

1. A user `exclude` no longer replaces the defaults

mergeConfig did exclude: override.exclude ?? base.exclude. Setting exclude at all inherited nothing from DEFAULT_CONFIG, and fell further behind whenever a release added a default pattern. This repository’s own config had done so: it was missing the 11 lockfile and tsconfig patterns added alongside the .json/.yaml includes, and reported pnpm-lock.yaml as a high large_file at 5,469 lines.

A second defect in the same area: crimes init hand-copied 9 of the 20 default exclude patterns, under a comment stating it “must never make crimes see less than it would with no config at all”. Every lockfile pattern was absent. The test suite named “generateConfig does not narrow the scan below zero-config” checks include in every one of its cases and exclude in none.

Changes:

exclude is additive to the defaults and deduplicated.
excludeDefaults: false restores wholesale replacement. This is the only way to un-exclude a path the defaults drop.
assets.exclude had the same contract and is now additive under the same flag. assets.include still replaces.
crimes init derives its list from DEFAULT_CONFIG.exclude.

This repository’s config drops from 28 patterns to the 8 that are repo-specific and scans identically (337 findings before and after).

2. `sync_io_in_hotpath` no longer charges one-shot scripts

On airflow, 227 of the detector’s 811 findings were in files carrying an if __name__ == "__main__": guard.

Three candidate signals had previously been tried and rejected, each because it also exempted task-sdk/src/airflow/sdk/execution_time/task_runner.py, which is production code. The third — “every same-file call path starts inside the guard, in a module nothing imports” — was recorded as the signal that would work. Tracing it in that file:

_send_error_email_notification ← finalize ← main ← guard
_handle_trigger_dag_run        ← run      ← main ← guard

finalize and run have no other same-file caller, and crimes reports the module at 0 direct importers, so the rule exempts both findings — the two the remediation doc records as correctly reported.

The signal used instead counts textual module references rather than import-graph edges. airflow launches the module with python -m, but the repository mentions it 42 times, including mock.patch("airflow.sdk.execution_time.task_runner.startup"), which a string literal carries and an import graph does not. A module is treated as a script only when it has a guard and zero references. Test-only references do not qualify it.

repo	`sync_io_in_hotpath`	after	of the detector	of the report
airflow	811	680	−16%	−1.32%
mlflow	402	347	−14%	−0.85%
pydantic	17	11	−35%	−1.49%

task_runner.py’s two findings are still reported. On airflow the suppressed findings are in scripts/ (105), dev/ (22) and devel-common/ (4).

The first implementation of the index was quadratic — roughly 4,700 modules against 4,700 files — and did not complete in ten minutes. Collecting each file’s references once and inverting the map is linear; airflow scans in 91s.

3. crimes honours a repository’s own tooling exclusions

pydantic’s pydantic/v1 is excluded by its ruff, coverage, pyright and codespell configuration and is regenerated by make update-v1. It accounts for 85 findings, 17.5% of pydantic’s report.

This makes a config file into a suppression mechanism, so three rules constrain the reader:

Named tables only. airflow’s pyproject.toml has exclude = ["*"] under [tool.hatch.build.targets.sdist] at line 589. A reader honouring any exclude key reports airflow as clean. There is an integration test for that shape.
Exact keys, not prefixes. [tool.coverage.report] skip_empty = true and [tool.uv] exclude-newer = "4 days" are a boolean and a duration. codespell’s skip is legitimately a comma-separated string, so rejecting all string values is also wrong.
A value the reader does not fully understand is dropped, and no path is excluded on its account.

A path is skipped only when two or more independent tools name it. Two keys of the same tool count once. Across the corpus this corroborates exactly one directory; mlflow’s ruff-only list and drf’s codespell-only list corroborate at 1 and are left alone.

pydantic  487 → 402   −85, −17.5%
airflow   unchanged, no warning
mlflow    unchanged, no warning

Every skipped file is reported, aggregated by the pattern that authorised it:

[files_excluded_by_tooling] pydantic/v1 — 26 files
  26 files under pydantic/v1 were skipped because this repository's own
  tooling excludes them. Findings in them are absent, not zero.
  → Set "honourToolingExcludes": false in crimes.config.json to scan it anyway.

[tool.mypy] is not supported: its exclude takes regular expressions where pyright’s takes globs.

Only the Python half shipped. .gitattributes linguist-vendored, tsconfig exclude and .eslintignore are not implemented.

4. `commented_out_code`’s two variants agree

The language-js variant always appended a block hash as its discriminator; the universal variant appended one only when a file held more than one block.

The conditional form is unstable. A lone block’s candidate hash was discarded, so when an unrelated second block appeared anywhere in the same file both findings gained discriminators and the first one’s fingerprint changed. A crimes ignore entry written against it stopped matching.

Unified toward always identifying a block. Churn measured before choosing: 43 findings this way, 67 the other way. On the corpus, 42 fingerprints were retired and 42 introduced; finding counts are identical on every repo and no other detector is affected.

The intrinsic half of the pair is unchanged (language-js ramps 0.48 + 0.04/statement to 0.72; the universal twin is a flat 0.35). It is a scoring change and needs its own baseline.

5. One intrinsic ladder, and a gate on cross-pack disagreement

An audit of every charge implemented in both language packs found that 7 of 8 disagree; only large_function matches. Three kinds:

different constants for the same shape (boolean_naming_drift, mixed_utc_local_methods);
different shape: circular_dependency and deep_import express no intrinsic on the universal side and take a flat declared default, while Python ramps. An 8-module Python cycle reaches 0.92; the equivalent TypeScript cycle is fixed at 0.45. Universal deep_import sits at 0.30, which is NEUTRAL_INTRINSIC;
conditions on one side only (direct_date’s naive-parse surcharge, sync_io_in_hotpath’s async-handler base, both Python-only).

The direction is inconsistent, so no single per-pack offset corrects it. The seven are reported and not changed — landing them together would make none of them attributable.

What did change is the mechanism. The formula was already common: 14 universal detectors inlined round(Math.min(base + (n-1)*step, cap)) while the Python slate factored it into a helper. There is now one intrinsicFrom, and a test that fails on any cross-pack ladder disagreement not listed with a reason. Verified findings-neutral across 5,164 findings on cal.com, hono and three fixtures: zero findings added, removed or rescored.

6. `DEPTH_FLOOR` re-centred; the deep population is now reported

mean_ndcg_deep averages the scenarios whose fixture emits at least DEPTH_FLOOR findings. That membership is an input to the metric and nothing reported when it changed.

Fixture 01 emitted 42 findings against a floor of 40 and carried 21 of 28 deep scenarios (75%). Removing three findings from it would drop 21 scenarios out at once, moving the headline 0.3530 → 0.4863: +0.1333 with no scoring change. The largest genuine movement recorded to date is +0.0089.

All nine stored baselines are unaffected — fixture 01 has emitted exactly 42 findings throughout — but that is because nothing had removed a finding from messy-ts-app in nine releases.

Fixture depths are [1, 3, 4, 5, 9, 13, 42, 55, 92, 99]: a 28-finding gap, with the floor 2 below the nearest deep fixture and 27 above the nearest shallow one. Every floor in [14, 42] selects the same four fixtures, so re-centring to 28 changes nothing measured:

mean_ndcg_deep  0.3530 → 0.3530   byte-identical
scored_deep     28     → 28       same scenarios
fixture 01 headroom  2 → 14

Two diagnostics are now printed on every run: deep-set composition with a cliff marker, and on --compare, delta_on_stable_set plus an explicit statement when the two headline numbers are not a before/after.

7. Three deep scenarios and one new fixture

3 of 28 deep scenarios labelled any detector un-suppressed in 0.23.0, and 11 of the 28 were referenced by no scenario at all. Two of those already fired on an existing deep fixture.

Fixture 14 is new because no fixture had a pyproject.toml, so neither evals:ranking nor the agent run could observe the tooling-exclude feature. It also carries [tool.hatch.build.targets.sdist] exclude = ["*"] and [tool.uv] exclude-newer, so it fails if the allowlist widens.

deep scenarios labelling a differentiated detector   3/28 → 5/30
of the 28 referenced by any scenario                 17   → 19
fixture 01 share of the deep aggregate               75%  → 70%

Nine detectors remain unreferenced: agent_permission_sprawl, config_drift, contract_drift, dependency_provenance_gap, duplicated_policy, finder_duplicate_filename, mock_saturation, pass_through_abstraction, unsafe_retry.

Correction, after release. This section first stated that none of the nine fires on an existing deep fixture. That was based on checking fixtures 02, 03 and 04 and not 01. Two of them do fire on fixture 01, which is deep: contract_drift (2 findings, both high, on src/api/state.ts) and dependency_provenance_gap (1). Those two need a scenario only; the remaining seven need fixture content.

Corrections to the backlog

Four entries were inaccurate about their own subject.

B’s percentages are shares of the detector’s findings, not of the report. All three reproduce exactly (227/811, 88/402, 7/19), but in report terms they are 2.29%, 1.36% and 1.44%. The backlog’s impact table placed these beside a whole-report percentage under one column heading.
The signal recorded as “the one that would work” does not. It exempts the counter-example it was written to preserve.
pydantic’s third exclusion is [tool.pyright], not mypy. There is no [tool.mypy] table in the file. The distinction matters because the two keys take globs and regexes respectively.
“A coverage.warnings[] entry per skipped path” contradicts the shipped contract, in which subject is documented as never a file path because it is the aggregation key. Taken literally it would emit 85 warnings on pydantic.

Eval baseline

Full matrix: 102 combinations (51 scenarios × 2 agents), up from 96 because this release adds three scenarios.

claude  0.82 → 0.84
codex   0.61 → 0.58

The scenario set changed, so those are not a before/after. On the 48 scenarios present in both runs:

agent	headline	stable 48
claude	+1.3pp	+0.0pp (0.823 → 0.823)
codex	−2.5pp	−5.1pp (0.608 → 0.557)

All 96 stable pairs received a byte-identical scan_context: no fixture’s findings moved in this release, and evals:ranking reported “no scenario moved” at every step. The agents saw the same scan output as at 0.24.0, so the run is a repeat sample and measures agent variance:

claude  16 of 48 scenarios moved with identical input, net  +0.0pp
codex   16 of 48 scenarios moved with identical input, net  −5.1pp

Codex’s movements run in both directions — bugfix-01-messy-ts-app 0/2 → 2/2 and bugfix-13-polyglot-plan-drift 0/4 → 2/4 against review-05-permission-and-parallel 4/7 → 0/7. The stated ±3pp band for codex does not cover a −5.1pp swing on identical input and should be re-derived.

structural_pass_rate matches detector ids in an agent’s response and cannot observe changes 1–3 above, which move findings on real repositories but not on the fixtures. The corpus measurements are the evidence for those; this run confirms no fixture behaviour changed.

Deterministic metric:

mean nDCG (deep)  0.3530 → 0.3498   headline
                          +0.0000   on the 28 scenarios deep in both runs

The first run attempt was interrupted at 73/102 and completed with --resume, which re-ran only the missing combinations. The directory reports no missing combinations and every result file postdates the build it was produced with.

Compatibility

schema_version stays 0.7.0. All schema changes are additive: one new coverage.warnings[].kind (files_excluded_by_tooling) and two new optional config keys (excludeDefaults, honourToolingExcludes).

Fingerprints move for one population: commented_out_code findings in single-block non-JS files, 42 across the corpus. Existing crimes ignore entries for those need re-adding.

Tests: 2,221 → 2,313.