🤓 Latest Submissions
Claim Check - does this PR do what it says?
We pointed Claim Check at a real, public pull request that four rounds of human review had already approved: KeeperHub #2476, 10 files, 1,752 added lines, 44,718 characters of review prose. Of the 29 falsifiable claims in its description, 28 held and one did not. The description promised an always-on SSRF guard on every outbound request; one connection-test path called the raw fetch global on an operator-supplied URL, with no guard. Four reviewers signed off, and the claim was still false. We fixed it the same night with six tests and disclosed it in the thread ourselves. That is the expensive half of code review. Reading the diff is the visible half; checking that every claim in the description is actually true in the code is where reviewers run out of time. Under load they skip it, and that is how a pull request whose description says it validates the payload gets merged without validating the payload. Claim Check reads a pull request the way a careful reviewer does, and parallelises the part a human cannot. It extracts every falsifiable behavioural claim from the description and the review thread, discarding intent and opinion. It then gives each claim to its own IBM Bob subagent, running concurrently with full-repository context, and each returns a verdict and a file:line citation. The output is a table: claim, holds or does not hold or cannot tell, evidence. It does not approve, reject or grade; it tells the reviewer which claims are backed by code and which ones nobody has checked. Why this needs Bob 2.0 and not a prompt: the claims are in prose (document understanding), the evidence is spread across files and survives a rebase (whole-repository context), and the work is N independent searches rather than one question (subagents as parallel tasks). Remove any one and the tool either misses claims or takes longer than reading the diff. Impact, measured on this PR: 29 claims found, 28 held with file:line evidence, 1 does not hold, 0 left unchecked; the tool agrees with 14 of the 14 conclusions the maintainer wrote by hand across four review rounds. Bob's subagents produced these verdicts during the build; the demo replays that real run offline because the hackathon budget is spent, so it is reproducible with no API calls. We also know its ceiling, and found it on the same codebase: two canonicalisation functions disagree, the signer drops undefined keys and the reviewed verifier does not. Byte-identical today, silently incompatible the first time a field becomes conditional. Four rounds of review missed it because it is invisible in any single file, and no claim mentions it, so no subagent looked. That is a boundary of the method, stated plainly.
27 Sep 2026
