RuleProof: Tested Rules for AI Coding Agents

Problem: Teams adopting an AI coding agent quickly learn that it does not know their unwritten rules: that shipped orders are final, that the audit module has no 'log()' function, that partial refunds round down. Reviewers catch these mistakes, and the agent makes them again next week. The usual fix is a rules file (AGENTS.md or '.bob/rules'), written by hand or generated with '/init'. Both are guesswork. Nobody measures which rules change the agent's behaviour, and every rule adds tokens to every future task. ETH Zurich's AGENTbench study found that LLM-generated context files did not improve task success but added over 20% to cost. The knowledge that matters lives in the team's history (repeated review comments, reverted commits, red CI runs) which no agent reads. Solution. RuleProof turns that history into rules that are proven to help, using IBM Bob end to end: 1. Mine: A custom Bob IDE mode, 'rule-miner', runs parallel explore subagents over review comments, git history and CI logs, and proposes rules only for mistakes raised in at least two places, each backed by exact quotes. 2. Verify: A deterministic checker rejects any rule whose quoted evidence does not appear verbatim in the sources. 3. Test: For each rule, Bob Shell performs the same coding task on throwaway copies of the repository, with and without the rule, three times, and a behavioural check scores every result. 4. Decide and ship. A rule is kept only if it measurably raises the pass rate; rules Bob already follows are flagged redundant. Kept rules are exported for '.bob/rules/', and a public dashboard shows each rule's evidence, pass rates and Bob's actual code with and without it. Results: On our sample service (75 real Bob runs), three of five mined rules took Bob from 0/3 to 3/3 passes; two were redundant because Bob already inferred them from the code. In a head-to-head across five tasks, Bob with no rules passed 20%, with Bob's own '/init' output 33%, and with RuleProof's three rules 100% - using a rule file seven times smaller than '/init's and at about 30% lower cost per run. With 15 runs each, the 95% confidence intervals (7-45%, 15-58%, 80-100%) do not overlap. '/init' captured only the one convention our tests had already marked redundant. On real projects, mining works too: from 727 anonymized review comments of three open-source libraries (ky, Express, Fastify), RuleProof mined 23 rules that reviewers enforced in at least two different pull requests, with 80 of 81 quotes verified. Bob's '/init' for the same repositories wrote 18 KB of rules yet fully captured only one of them and missed nineteen. Who it is for: tech leads and teams adopting IBM Bob. They run RuleProof on their repository, review each mined rule with its evidence and measured effect in the dashboard, and download the proven rules into .bob/rules: fewer repeated review comments, less rework, and small rule files in which every rule has earned its place. Live demo: https://ruleproof.vercel.app