🤓 Latest Submissions
Plumbline: is your test plan still true?
Before a release, someone signs a checklist saying every case in the test plan has a passing test and the spec matches the code. Nobody really checks. I once shipped a project whose CI badge was green for days while it ran no tests at all. Plumbline is a custom mode and skill for IBM Bob that does the checking. Bob reads the team's real documents (an .xlsx test plan, a .docx spec, a .pdf release checklist), turns every row, clause and tick into a claim, and finds the test behind each one. Then, for every case with a test, a subagent that never sees the test changes the code so the claim is false, and writes a witness proving the break. If the test still passes, it's a test in name only. Spec clauses and checklist ticks are judged against the code and git history. The result is one HTML report. Who it's for: release owners and QA leads who sign releases off, and developers who want to know which tests actually protect them. They copy .bob/ into a repo, switch Bob to the Plumbline mode and ask for an audit. What's different: coverage says a line ran; mutation testing breaks code at random and never reads the plan. Plumbline starts from what the team promised, and never accuses a test without proof. On a sample service whose defects I recorded in an answer key before Bob saw it: the plan claims 84 passing cases. Bob found tests for 60, and 8 of those pass on broken code (2 the key had missed). 4 of 24 spec clauses and 4 of 10 checklist ticks are false. Bob agreed with the key on 83 of 84 mappings and on every clause and tick. A free, no-AI checker does the same for any public README: 23 of the 100 most-starred organisation repos on GitHub fail it.
27 Sep 2026
