FlakeHunter: Find & Fix Flaky Tests with IBM Bob

**The problem.** A flaky test passes and fails on the same code. Nothing changed, yet CI turns red. Developers click "re-run", wait, and move on. Every team knows the cost: wasted CI minutes, blocked merges, and worst of all, lost trust. Once people assume red means "probably flaky", real bugs slip through. Fixing flakiness by hand is slow detective work, because the failure usually disappears the moment you try to reproduce it. The root causes repeat: timing assumptions, test-order dependency, shared state, unseeded randomness and unreliable network calls. **The solution.** FlakeHunter turns that detective work into one command, `flakehunter run`, or one button in the web UI: 1. **Detect.** It runs the whole suite N times in random order. Any test that both passes and fails is flaky, proven by re-runs rather than guessed. 2. **Hint.** A static AST scan adds clues such as timing, random, network or shared-state. Hints only guide; they never decide. 3. **Diagnose and fix with IBM Bob, in parallel.** For each flaky test, FlakeHunter starts a headless IBM Bob agent (`bob run`) in its own isolated copy of the project. Bob reads the test and the code it uses, names the root cause, explains it in two sentences, and makes a minimal fix to the test file. 4. **Verify.** Each fix is applied and its test file is re-run 50 times in random order. A fix is kept only at 50/50. Otherwise the original file is restored byte-for-byte and the fix is kept as a suggestion. 5. **Prove and report.** The whole suite runs again, and a self-contained HTML report shows before/after numbers, Bob's root causes and explanations, the diffs, and estimated CI time saved. **Safety by design.** Bob never works in the real project, only in temp copies. If Bob edits anything other than the test file, the fix is rejected. Each Bob call has a cost cap and a timeout. Nothing is kept unless it is proven. **Result.** On our demo shop app (15 tests, 4 of them flaky), CI was fully green in only 15% of runs with no code changes. In a single run, IBM Bob diagnosed all four tests in parallel: shared state leaking between tests, a random sleep exceeding an SLA, a simulated payment-gateway timeout, and an unseeded random choice. All four fixes passed 50/50 verification. Afterwards: 0 flaky tests and 100% green runs. With conservative assumptions (10 CI runs a day, 5 minutes per re-run), that is about 25 hours of CI time saved per month for this small suite alone. **Why it matters.** FlakeHunter doesn't just suggest code. It closes the loop from detection to diagnosis, fix and proof, the steps a senior engineer would take, and it does them in parallel and in minutes. The live demo runs detection and verification for real on the server, replaying the recorded real IBM Bob run so the public page needs no API key.