TestPilot: Test-Gap Hunting with IBM Bob

PROBLEM Test debt builds up quietly. Coverage slips file by file, deprecation warnings pile up between releases, and some parts of an app never get a test runner at all. Paying it back means measuring the gaps, writing many small tests, updating outdated APIs and reporting what changed. It is slow, repetitive work, so teams keep postponing it until something breaks. SOLUTION TestPilot packages that whole developer workflow into IBM Bob so it runs from one prompt and gives a repeatable, measurable result: - A custom Bob mode, "TestPilot", acts as a test-and-maintenance engineer. Its first rule: never change app behaviour just to make a test pass. - A reusable Bob skill, "test-gap-hunter", runs a 5-step workflow: Measure (tests + coverage + warnings), Plan (rank gaps by risk), Fix (write tests, update outdated code; backend and frontend can go to separate Bob subagents), Verify (re-run everything; if the app is wrong, report a bug instead of bending the test) and Report (before/after tables in docs/RESULTS.md). - A scan script and a dashboard show the results, with a live "Run scan" button that re-runs all 102 tests on screen. Light/dark mode, works on phones. - A GitHub Actions guard fails any change that drops coverage below 95% or brings a deprecation warning back. TARGET USERS & HOW THEY USE IT Developers and tech leads maintaining existing apps. Open the repo in Bob, pick the TestPilot mode, send "Use the test-gap-hunter skill on <folder>", review Bob's changes, and check the dashboard. CI keeps the gains from slipping. RESULTS (IBM's open-source Galaxium Travels app, FastAPI + React) - Backend coverage 86% -> 95%; server.py 59% -> 100%; backend tests 29 -> 46 - Deprecation warnings 6 -> 0 (Pydantic v2 ConfigDict, SQLAlchemy 2.0 import, timezone-aware datetime) - Frontend: Vitest + React Testing Library set up from zero; tests 0 -> 56 - 2 real bugs found and reported instead of hidden (users saw "NaNh NaNm" for invalid flight times; an error check returned null instead of false), then fixed in a fresh TestPilot run - Every number is re-checked by GitHub Actions on each push (passing). WHAT MAKES IT DIFFERENT It is honest by design: it separates "the test is wrong" from "the app is wrong", reports real bugs, and backs every claim with a measured before/after number instead of just generating code. Links: code https://github.com/pacyuzu16/testpilot | live dashboard https://pacyuzu16.github.io/testpilot/