crimes v0.24.0 — the ceiling becomes a scale
Theme: the other half of 0.23.0. That release proved
STRUCTURAL_CEILING’s stated rationale false and deliberately did not
touch the mechanism, so the input fix stayed attributable. This changes
the mechanism.
before Math.min(scored, STRUCTURAL_CEILING)after round(scored * STRUCTURAL_CEILING)schema_version stays at 0.7.0. No field is added, renamed or
retyped, and no finding is added or removed anywhere on the corpus.
Scores and ordering move.
A clamp does not rank; it hands ranking to something else
Section titled “A clamp does not rank; it hands ranking to something else”Structural findings — “this file is big”, “this function is long” — are capped so they cannot outrank a finding that says an agent will be actively misled. The cap was a hard clamp, and a hard clamp throws away everything above it.
Measured at 0.22.0: it collapsed 760 of zulip/zerver’s 1505
findings onto exactly 0.30, from 31 distinct pre-clamp levels spanning
0.31–0.63. Across the corpus the plateau covered 22.8%–61.4% of a
whole report.
That is worse than a tie, because
rank_score = agent_risk * (1 + recency * 0.5)With half a report sharing one agent_risk, the order of that half was
decided by recency — how recently the file was touched — and then
by severity, confidence and file path. Half of pydantic’s report was
sorted by file age.
What changed
Section titled “What changed”Scaling instead of clamping keeps the class ordered by its own severity
while still mapping onto [0, CEILING], so a length finding still
cannot outrank a differentiated one.
| repo | findings at exactly 0.30 | distinct agent_risk values |
|---|---|---|
| mlflow | 2778 → 46 | 57 → 63 |
| zulip/zerver | 777 → 4 | 42 → 48 |
| pydantic | 296 → 4 | 43 → 46 |
| hono | 99 → 4 | 43 → 49 |
| drf | 57 → 0 | 17 → 21 |
Resolution rises on every repo — the ranking says more, not less.
Length findings stop leading
Section titled “Length findings stop leading”The clamp never managed this, which is the evidence that it was not doing the job its own comment claimed:
| repo | top-20 structural | top-50 structural |
|---|---|---|
| pydantic | 6 → 0 | 23 → 0 |
| drf | 15 → 10 | 45 → 34 |
| zulip/zerver | 0 → 0 | 14 → 0 |
| hono | 0 → 0 | 1 → 0 |
drf stays structural-heavy because it is — 72 of its 88 findings are structural, so its head cannot be anything else.
On pydantic the top-5 file set is unchanged and the order moves:
core_schema.py (19 findings, 16 of them low, all length) goes 2nd →
5th, while mypy.py and fields.py, which carry high-severity
differentiated findings, move up. The clearer effect is within a file
— _generate_schema.py’s God Functions now rank by their own size and
nesting (121 lines at depth 7 first) rather than by the tiebreak they
fell through to when 296 pydantic findings shared one score.
Measured, from the 0.23.0 baseline
Section titled “Measured, from the 0.23.0 baseline”R5’s numbers for this change were taken against 0.22.0, with 28
detectors still suppressed. They are not evidence about the squash on
top of the intrinsics fix, so the measurement was re-run rather than
reused.
Deterministic (evals:ranking — no agent, no noise band), deep
fixtures:
| deep bucket | n | mean nDCG | up | down |
|---|---|---|---|---|
| no structural expectation | 19 | 0.3668 → 0.3757 (+0.0089) | 11 | 0 |
| priority IS structural | 7 | 0.3586 → 0.3382 (−0.0205) | 0 | 5 |
Both columns unanimous. The headline deep mean is 0.3538 → 0.3530 —
essentially flat, because the two buckets net out — and all is
0.4926 → 0.4799. The second bucket is the length-labelled scenarios §28
has already disowned, plus their neighbours: their labelled right answer
is a length finding, and the product has decided length findings
should not lead.
Concentration does not worsen. mlflow’s dominant-type lift falls 2.59 → 2.34, zulip holds at 1.20, hono at 2.80, and pydantic’s head becomes measurable at 1.66 where it previously had no agent-signal dominant type at all.
Eval baseline
Section titled “Eval baseline”| agent | 0.23.0 | 0.24.0 | move | noise band |
|---|---|---|---|---|
| claude | 0.81 | 0.82 | +1pp | ±6pp |
| codex | 0.63 | 0.61 | −2pp | ±3pp |
Both inside their bands, and that is the expected result.
structural_pass_rate matches a detector’s literal id in the agent’s
response text, so it cannot see a re-ranking at all — an agent quotes
the right id whether the finding ranked 1st or 30th. That blindness is
precisely why evals:ranking was built, and this release is a pure
re-ranking: no finding is added, removed or rescored in severity
anywhere on the corpus.
Read the deterministic split above. A flat result here is not evidence
the change did nothing; it is evidence the instrument is the wrong one
for this kind of change, which evals/README.md states directly.
The part that is a trade
Section titled “The part that is a trade”The two effects are inseparable at 2-decimal precision. The
structural band has 31 slots (0.00–0.30) and the input has 101
levels, so a monotonic map cannot re-spread the clamped tail without
also lowering the rest of the class. This release does not only preserve
order — it also pushes the whole structural class down, so a typical
large_function lands near 0.05–0.09 rather than 0.16–0.30.
That was measured and accepted, not overlooked. It is why the structural-labelled bucket drops, and it is the thing to reopen if you think length findings should sit higher.
2,218 → 2,221. The three new ones pin the mechanism: order preserved within the structural class, the ceiling still respected across an input grid, and the agent-signal class left untouched.
The two pre-existing ceiling tests could not do this. Both use maximal
inputs — intrinsic, churn, test_gap and blast_radius all 1 —
which is the single point where a clamp and a squash agree. That is why
a mechanism change touching 22–61% of every report passed the entire
suite untouched when it was first tried.
Still unsettled
Section titled “Still unsettled”The level 0.3 remains unvalidated: correcting a mechanism does not
validate a constant, and nothing here chooses that number. Also still
open — the class table is hand-maintained and standard has zero
members across all 70 detectors; the two language packs disagree about
sync_io_in_hotpath (0.55 JS, 0.50 / 0.70 Python); and 41 intrinsics
remain literals inside their own detectors rather than in
INTRINSIC_DEFAULTS.
Full evidence in
docs/calibration-followups.md
§ “0.24.0 — the ceiling becomes a scale”.