FlakyTestHunter

Problem Flaky tests — tests that pass or fail inconsistently with no underlying code change — are a quiet but expensive problem in software engineering. They erode trust in CI pipelines until "just rerun it" becomes standard practice, they hide genuine bugs behind statistical noise, and diagnosing them usually means a developer manually re-running a suite dozens of times while guessing at root causes from stack traces. There's no standard tooling that treats flakiness itself as a first-class problem to detect, explain, and fix. Solution FlakyTestHunter automates the entire flaky-test workflow using a 3-stage IBM Bob 2.0 subagent pipeline: Detector — runs the target test suite 10 times in a row and reports the raw pass/fail pattern, flagging any test with inconsistent results. Diagnosis — reads the actual source code behind the failing test and explains, in plain English with exact line references, the root cause of the non-determinism (unseeded randomness, timing dependencies, etc.). Fix & Proof — applies a deterministic fix (typically mocking the non-deterministic source), then re-runs the suite 10 more times to prove the fix works, reporting a final pass/fail ratio. Every stage is a genuine, live Bob Shell task — invoked headlessly via bob run --accept-license from a FastAPI backend, with zero human approval needed mid-run. Nothing is scripted or faked. What makes this a complete, working product (not just a proof of concept): Live deployed backend on Render, running inside a Docker container with Node.js, Python, and the Bob Shell CLI all installed non-interactively — solving a genuine installer/PATH challenge to get Bob running unattended in a cloud container. Live deployed frontend on Netlify, offering three ways to see the pipeline in action: an instant cached demo (real, previously-captured Bob output), a live "Run Live" button that triggers a fresh Bob run on the deployed backend in real time (~1-2 minutes, real Bobcoins spent), and an Upload & Fix feature where anyone can upload their own Python file and watch Bob diagnose and fix it live, with the corrected file downloadable immediately. Proven generalization: we validated the pipeline not only against our seeded demo app (a function with an intentional ~33% failure rate from unseeded random.uniform), but also against a completely different, previously-unseen flaky pattern (time/latency-based) submitted through the live upload feature — confirming the approach isn't overfit to one example. Real, transparent cost data: every run reports its actual Bobcoin cost (typically ~0.3–0.7 Bobcoins per full pipeline), all well within the 40-Bobcoin trial budget, and all of it is visible in the raw Bob session output shown directly on the site. Full transparency: 10 real, unedited screenshots from actual Bob Shell terminal sessions are included in the repo and displayed in-app, so nothing about the Bob usage is staged or summarized after the fact. FlakyTestHunter is built for any engineering team that has flaky tests sitting in their CI right now and wants an AI reviewer that doesn't just flag the problem, but explains it and proves the fix.