Skip to content

crimes v0.24.0 — the ceiling becomes a scale

Theme: the other half of 0.23.0. That release proved STRUCTURAL_CEILING’s stated rationale false and deliberately did not touch the mechanism, so the input fix stayed attributable. This changes the mechanism.

before Math.min(scored, STRUCTURAL_CEILING)
after round(scored * STRUCTURAL_CEILING)

schema_version stays at 0.7.0. No field is added, renamed or retyped, and no finding is added or removed anywhere on the corpus. Scores and ordering move.


A clamp does not rank; it hands ranking to something else

Section titled “A clamp does not rank; it hands ranking to something else”

Structural findings — “this file is big”, “this function is long” — are capped so they cannot outrank a finding that says an agent will be actively misled. The cap was a hard clamp, and a hard clamp throws away everything above it.

Measured at 0.22.0: it collapsed 760 of zulip/zerver’s 1505 findings onto exactly 0.30, from 31 distinct pre-clamp levels spanning 0.31–0.63. Across the corpus the plateau covered 22.8%–61.4% of a whole report.

That is worse than a tie, because

rank_score = agent_risk * (1 + recency * 0.5)

With half a report sharing one agent_risk, the order of that half was decided by recency — how recently the file was touched — and then by severity, confidence and file path. Half of pydantic’s report was sorted by file age.

Scaling instead of clamping keeps the class ordered by its own severity while still mapping onto [0, CEILING], so a length finding still cannot outrank a differentiated one.

repofindings at exactly 0.30distinct agent_risk values
mlflow2778 → 4657 → 63
zulip/zerver777 → 442 → 48
pydantic296 → 443 → 46
hono99 → 443 → 49
drf57 → 017 → 21

Resolution rises on every repo — the ranking says more, not less.

The clamp never managed this, which is the evidence that it was not doing the job its own comment claimed:

repotop-20 structuraltop-50 structural
pydantic6 → 023 → 0
drf15 → 1045 → 34
zulip/zerver0 → 014 → 0
hono0 → 01 → 0

drf stays structural-heavy because it is — 72 of its 88 findings are structural, so its head cannot be anything else.

On pydantic the top-5 file set is unchanged and the order moves: core_schema.py (19 findings, 16 of them low, all length) goes 2nd → 5th, while mypy.py and fields.py, which carry high-severity differentiated findings, move up. The clearer effect is within a file — _generate_schema.py’s God Functions now rank by their own size and nesting (121 lines at depth 7 first) rather than by the tiebreak they fell through to when 296 pydantic findings shared one score.

R5’s numbers for this change were taken against 0.22.0, with 28 detectors still suppressed. They are not evidence about the squash on top of the intrinsics fix, so the measurement was re-run rather than reused.

Deterministic (evals:ranking — no agent, no noise band), deep fixtures:

deep bucketnmean nDCGupdown
no structural expectation190.3668 → 0.3757 (+0.0089)110
priority IS structural70.3586 → 0.3382 (−0.0205)05

Both columns unanimous. The headline deep mean is 0.3538 → 0.3530 — essentially flat, because the two buckets net out — and all is 0.4926 → 0.4799. The second bucket is the length-labelled scenarios §28 has already disowned, plus their neighbours: their labelled right answer is a length finding, and the product has decided length findings should not lead.

Concentration does not worsen. mlflow’s dominant-type lift falls 2.59 → 2.34, zulip holds at 1.20, hono at 2.80, and pydantic’s head becomes measurable at 1.66 where it previously had no agent-signal dominant type at all.

agent0.23.00.24.0movenoise band
claude0.810.82+1pp±6pp
codex0.630.61−2pp±3pp

Both inside their bands, and that is the expected result. structural_pass_rate matches a detector’s literal id in the agent’s response text, so it cannot see a re-ranking at all — an agent quotes the right id whether the finding ranked 1st or 30th. That blindness is precisely why evals:ranking was built, and this release is a pure re-ranking: no finding is added, removed or rescored in severity anywhere on the corpus.

Read the deterministic split above. A flat result here is not evidence the change did nothing; it is evidence the instrument is the wrong one for this kind of change, which evals/README.md states directly.

The two effects are inseparable at 2-decimal precision. The structural band has 31 slots (0.000.30) and the input has 101 levels, so a monotonic map cannot re-spread the clamped tail without also lowering the rest of the class. This release does not only preserve order — it also pushes the whole structural class down, so a typical large_function lands near 0.05–0.09 rather than 0.16–0.30.

That was measured and accepted, not overlooked. It is why the structural-labelled bucket drops, and it is the thing to reopen if you think length findings should sit higher.

2,218 → 2,221. The three new ones pin the mechanism: order preserved within the structural class, the ceiling still respected across an input grid, and the agent-signal class left untouched.

The two pre-existing ceiling tests could not do this. Both use maximal inputs — intrinsic, churn, test_gap and blast_radius all 1 — which is the single point where a clamp and a squash agree. That is why a mechanism change touching 22–61% of every report passed the entire suite untouched when it was first tried.

The level 0.3 remains unvalidated: correcting a mechanism does not validate a constant, and nothing here chooses that number. Also still open — the class table is hand-maintained and standard has zero members across all 70 detectors; the two language packs disagree about sync_io_in_hotpath (0.55 JS, 0.50 / 0.70 Python); and 41 intrinsics remain literals inside their own detectors rather than in INTRINSIC_DEFAULTS.

Full evidence in docs/calibration-followups.md § “0.24.0 — the ceiling becomes a scale”.