Dogfooding round protocol: 0.14 → 0.17

Date: 2026-08-02 Build under test: crimes 0.17.0, schema_version 0.4.0, built from source at packages/cli/dist/index.js. Shim: ~/crimes-dogfood/bin/crimes

This file is the contract. It exists so that every repo in the corpus is measured the same way, by whoever measures it.

The question

Not “is this finding correct” but “would a maintainer of this repo act on it”. Precision falls out as a by-product; the headline number per detector is its act rate, and that is what decides whether a detector is kept, retuned, gated, or sunset.

Surface under test

Never run outside its own fixtures before this round:

release	surface
0.14.0	Python pack: 7 detectors, tree-sitter WASM parser
0.15.0	cross-language pack: 3 detectors + coverage.by_package
0.16.0	10 detectors: correctness, cross-file authority, agent hygiene
0.17.0	large_file docs shape; discriminator fingerprints

The other ~45 core detectors are in scope opportunistically.

Boundaries: these are not negotiable

1. Do not change detector behaviour, thresholds, or scoring during the round. Tuning mid-corpus means repos scanned before and after were measured with different instruments, and the round is worthless. Record the problem and keep going.
2. Do not write anything into a corpus repo. No crimes init, no .crimes/ directory, no config, unless the step explicitly says to and the artefact is captured and then reverted.
3. Never scan from inside the crimes working tree, and never clone a corpus repo into it.
4. Judging is not done in this phase. Capture and observe. Verdicts come later, from a blind judge.

Per-repo procedure

Let $R be the repo path and $N its short name.

crimes=~/crimes-dogfood/bin/crimes
out=~/crimes-dogfood/artifacts
log=~/crimes-dogfood/logs

# 0. pin it
git -C "$R" rev-parse HEAD

# 1. the JSON contract — timed
/usr/bin/time -p $crimes scan "$R" --all --format json --no-init \
  > "$out/$N.json" 2> "$log/$N.time"

# 2. the human view — this is what a user actually sees
$crimes scan "$R" --no-init > "$out/$N.human.txt" 2>&1

# 3. the other commands, each timed and captured
$crimes hotspots "$R" --no-init        > "$out/$N.hotspots.txt" 2>&1
$crimes verdict --no-init              > "$out/$N.verdict.txt" 2>&1   # run with cwd=$R
$crimes context <a-real-file> --no-init > "$out/$N.context.txt" 2>&1
$crimes explain <an-id> --no-init      > "$out/$N.explain.txt" 2>&1
$crimes diff <range> --no-init         > "$out/$N.diff.txt" 2>&1

Mandatory robustness checks

A degraded scan and a clean scan look identical: indexes.ts swallows index-build failures by design. So for every repo, record from the JSON:

• coverage.files_total, coverage.files_by_language, coverage.files_universal_only
• coverage.packs_loaded: and whether the pack actually produced findings. packs_loaded reports what was loaded, not what ran.
• coverage.by_package where present (0.15 surface)
• anything on stderr, any non-zero exit, any wall-clock outlier
• for Python repos: whether modern syntax parsed: 3.11/3.12 generics, match, walrus, except*. Check by confirming findings exist in files known to use them, not by absence of an error.

The zero case

For a single-language repo, the three cross_language_* detectors must produce zero findings, and the Python detectors must produce zero on a repo with no .py. A false positive here is worse than a missed finding, because it means packs_loaded cannot be trusted. Record this explicitly per repo, as a positive statement, not silence.

The friction log

Written during, never reconstructed after. One file per repo at ~/crimes-dogfood/logs/$N-friction.md, appended as things happen.

Record anything that made the tool worse to use, however small:

• output that was confusing, buried the lede, or was too long to read
• a flag or command you reached for that does not exist
• a finding whose summary did not explain itself without opening the file
• a crash, a hang, a stack trace, a wrong exit code
• evidence that was not reproducible across two runs
• anything that would make you stop using the tool

Format: one entry per observation:

### <short title>
- **repo:** <name>
- **command:** <exact command>
- **what happened:** …
- **why it matters:** …
- **severity:** annoyance | real problem | blocker

Negative results belong here too: a detector that fired zero times across a repo where you expected it to fire is an observation about the detector.

Sampling rule for judging

Stated up front, then followed. Deviating silently is how a round produces a flattering number instead of a useful one.

Every finding from a 0.14+ detector (the Python pack, the three cross-language detectors, the ten 0.16 detectors), plus the top 20 findings by agent_risk per repo.

If that exceeds the budget for a repo, sample randomly within the rule and record the seed and the sample size: never by picking.

Judging protocol

The judge sees a card and nothing else:

• repo name, file path, line range
• the code, ±30 lines around the finding
• the finding’s summary and evidence

The judge does not see: the detector name, detector_id, charge, any score, or any indication that a tool produced it.

Question put to the judge:

Would a maintainer of this repository act on this? Answer act, good-to-know, or noise, and give one line of reasoning.

Two independent judges: codex, blind, and the session agent, from the same card. Disagreements are recorded, not resolved away: a finding two readers split on is itself evidence the finding is ambiguous. The adjudication and its reason are part of the deliverable.

Verdicts are then written through crimes feedback, so the round exercises the feedback loop as well as measuring it.

Deliverables

1. docs/dogfooding/2026-08-02-0.14-to-0.17.md: the report
2. docs/dogfooding/2026-08-02-log.md: merged friction log
3. A sunset shortlist: every detector with an act rate low enough to question its existence, with a recommendation
4. Ranked recommendations, each labelled with its cost: config default / detector tuning (patch bump + eval re-run) / schema change (minor + migration)
5. Detectors that fired zero times across the entire corpus, listed explicitly. A detector that never fires in the wild is a finding about the detector.

Recommendations are not applied in this pass. They come back as a decision.