🤓 Latest Submissions
Counterexample
The problem AI now writes pull requests faster than any team can review them. IBM's own research shows 85% of DevSecOps professionals agree AI has shifted the bottleneck from writing code to reviewing and validating it. Green CI has become a false signal of safety: AI-generated tests often confirm what AI-generated code does, bugs included, because the same reasoning that wrote the bug also wrote the test that doesn't catch it. Reviewers are left re-reading every line to reconstruct intent that was never verified in the first place. The solution Counterexample is a PR verification tool that replaces reviewer opinions with executed evidence. For any pull request, it runs two independent, complementary checks: Mutation testing. Counterexample parses the diff, injects small deliberate bugs (flipped comparisons, swapped arithmetic, altered boolean logic) into only the changed lines, and reruns the PR's own test suite against each mutant. If the tests don't catch the injected bug, that's a real gap in test coverage — a mutation score below 100% means the tests aren't actually proving what they claim to. Claim falsification, via IBM Bob 2.0. Counterexample extracts the concrete, testable claims a PR makes — from its diff and its linked issue — then spawns a parallel subagent per claim, using Bob 2.0's subagent architecture. Each subagent writes and actually runs one adversarial test designed to break that specific claim, reporting FALSIFIED (with the failing test and real output) or HELD. Why two layers, not one — proven, not assumed We validated this design on a real PR: a "stacked coupons" checkout feature with a planted logic bug. Mutation testing alone scored a reproducible, genuine 100% — LOOKS SOLID. The bug is a wrong-variable-reference error, a class no operator-level mutation can express, so mutation testing legitimately cannot catch it alone. Claim-level verification, run through Bob, found it immediately — and found more than we planted. Of five claims extracted from the PR and its linked issue, four were falsified with real executed evidence: the percent discount computed against the wrong subtotal, a deleted zero-floor clamp letting totals go negative, order-dependence masking the same underlying bug, and a silent breaking change for every existing caller using the old API. Two of these four bugs were not intentionally planted — Counterexample found them on its own. The output One unified Review Receipt combines both signals: mutation score and claim verdicts, side by side, so a reviewer sees both "would the tests catch an injected bug" and "does the PR actually do what it claims" in a single artifact. A GitHub Action runs the mutation-testing layer automatically on every PR; the claim-verification layer runs on demand through Bob, since it requires the deeper reasoning only an agentic reviewer provides. CI said ship it. Counterexample said here's the bug — with proof.
27 Sep 2026
Bimanual VLA Table-Setting: Measured, Not Assumed
A dual-arm SO-101 table-setting system for the Intel Bimanual VLA Manipulation track, where nearly every capability exists because a failure was measured and diagnosed before being fixed. The gripper initially couldn't pick anything up. Not a tuning problem, but a MuJoCo convex-hull artifact: the finger's collision hull is a 21.7° ramp across the grasp region, and no grasp height could fix a ramp that moves with the tool frame. The fix was explicit jaw pads fitted to the finger's real flat faces. An in-air handover between the arms turned out to be geometrically impossible on this platform: the gripper hull is wider than the longest grasp feature in the scene, even at the best pose found by exhaustive search. So the handover is table mediated, and that's documented, not assumed. Scripted pick success went from 57.5% to 92.5% through three measured fixes. A trained ACT policy (44k steps, 134 demonstrations) initially placed only 3 of 40 props and fixated on one object. Diagnosis: no language conditioning in stock ACT, a collapsed VAE latent, and class imbalance. Conditioning was added using ACT's existing spare observation slot. A second dataset with randomized starting states isolated one cause: conditioning now measurably redirects attention toward the named object, while the two
16 Sep 2026
The claim intake agent that refuses to guess
Claim intake is a form filled in by speech recognition. When the recognizer mishears a policy number, the claim lands on the wrong file, and nobody finds out until someone is denied. The failure is silent, because the agent has no way to know it was wrong. This agent splits the job. It listens and proposes. It does not decide. Every value goes through a server-side validator that checks it against the real policy data and returns one of three verdicts. Accepted, and it moves on. Rejected, and it asks again. Unconfirmed, and it reads the value back and waits for the caller to agree. The agent speaks the validator's exact words. It never invents a spelling. I tried three times to fix the recognizer instead. Transcription mode, turn-detection patience, and tool format hints were each measured against the same four policy numbers, and all three were null. The recognizer loses the D in KD4-1188 the same way in every configuration. That failure is acoustic, and no setting reaches it. Then I made it worse. AssemblyAI lets you bias the recognizer toward known terms, so I fed it the list of valid policy numbers. Accuracy improved and it started manufacturing false accepts. On one call the transcript's first reading was C411 and its final answer was KD4-1188, a real policy belonging to someone else, which then passed exact-match validation perfectly. The validator rested on an assumption I had never written down. The transcript is supposed to be an independent observation of what the caller said. Biasing the recognizer toward the answer key destroyed that, and every guard downstream stayed intact and stopped working. I removed the biasing. Policy numbers now require spoken confirmation whatever the match quality, because an exact match was never as strong as it looked. A reviewer sees every attempt, not just the ones that succeeded, each linked to what the caller said and when. The agent can be wrong. The record cannot be silently wrong.
30 Sep 2026
