🤓 Latest Submissions
TestSkeptic: Find What Passing Tests Miss
Passing tests can hide missing business-rule checks. In our shipping example, tests for order subtotals of 50, 100, and 150 all pass even when the free-shipping threshold is incorrectly lowered from 100 to 99. Developers need evidence that their tests can detect meaningful behavior changes. TestSkeptic is a Python proof of concept for developers and code reviewers investigating test gaps in small business rules. The user selects a test suite and runs one command. The tool first requires a passing baseline, then creates a fresh temporary copy for each of three predefined mutations and runs the same tests against it. It generates an HTML report showing KILLED, SURVIVED, or INVALID results alongside the actual test output. The original business code remains unchanged. The developer takes a surviving mutation to IBM Bob for repository-aware analysis, reviews the proposed test, enters it manually, and reruns the experiment. Separate before and after suites make the demonstration reproducible without deleting or restoring tests during the demo. In the demonstrated improvement, one additional test asserting shipping_fee(99) == 10 increased detection from two of three selected mutations to three of three. All four tests still passed against the original code. A separate malformed-mutation experiment produced a syntax error and was classified as INVALID rather than being counted as successful detection. Mutation testing is an established technique. This project's focus is an accessible workflow connecting controlled experiments, Bob-assisted reasoning, human review, and execution-based verification. It automates the repeated copy, mutate, run, classify, and report steps, helping developers focus on a concrete missing case. The current prototype supports one sample project and three predefined text replacements. It uses Python's standard library and requires no third-party packages. These results do not establish overall test coverage or prove the software is bug-free. Broader productivity benefits have not yet been measured.
27 Sep 2026
