Rashomon

Software teams spend most of their time reading code — about 58% of it (Xia et al., TSE 2018) — and almost all of our tooling measures the wrong side of that. We count tokens, lines and cyclomatic complexity: over 120 readability metrics have been proposed and Scalabrino et al. showed they fail to track human judgement. The newest idea, asking an LLM "is this readable?", is worse: CoReEval found the model's ratings correlate with humans at roughly ρ ≈ 0. Opinions, all the way down. **The insight: don't ask, check.** Predict what the function returns, then run it. A wrong prediction is not a style complaint — the code genuinely misled a reader. Five readers who disagree with each other are five readings of the same source, and the executing program decides who was right. **Pipeline.** Rashomon picks functions from a diff or by git churn, generates probe inputs from each signature, and executes them in a subprocess with a timeout to record ground truth. Five independent readers — `careful-stylist`, `speed-reader`, `regex-allergic`, `api-historian`, `edge-case-hunter` — see only the source and the inputs, never the outputs, and each returns a prediction per probe. Every prediction is graded against execution-derived rules, with abstentions counted as wrong. Separately, `git log` plus a source-diff filter counts how often each function has actually been fixed. The result is a misread rate, a disagreement count and a bug-fix history for every function. **Users.** Developers and reviewers, especially teams whose code is edited by AI agents: an agent's rewrite that is easier for five readers to predict is a rewrite that is easier for the next human *and* the next model to maintain. **How it is used.** In Bob, mode `rashomon` runs the skill that fans out the five readers as isolated subagents. `make gate` turns a run into a non-blocking PR comment that fails only when a function gets more confusing. Mode `clarifier` rewrites one confusing function, and `equiv` proves the copy behaves identically before the new reading is measured. The static leaderboard ranks functions by misread rate and overlays bug-fix counts. **Novelty.** CRUXEval uses code to grade models; Rashomon inverts it: independent readers, ground truth, disagreement metric — a lab study turned into a CI check that runs on every pull request. **Measured results (bundled demo, recorded session).** 9 functions, 5 readers, 135 graded predictions, 21 wrong (16%). `parse_duration` was misread by 7 of 15 readers (47%) while `mean` was read by all 5 — the ranking separates the confusing code from the clear code with no human label anywhere. **Live run on `boltons` (30 functions).** Five independent `gemini-3.5-flash-lite` readers, 410 graded predictions: 9 functions are *Consensus misread* (every reader wrong), 4 *Scattered* (readers disagree), 17 *Clear* (all readers clean). Side-effecting APIs like `iteritems` are misread 100% of the time. Misread rate correlates with real bug-fix history — Spearman ρ = 0.394, p = 0.031, 95% CI [0.03, 0.71] — so the score tracks maintenance pain, not style opinions.