Witnessed

Problem. Teams now let coding agents open pull requests, and a green test suite is the usual signal to merge. A suite can pass while the code the agent changed never ran. Line coverage cannot say which function in this change nobody has seen running. We ran Witnessed on the 44 merged pull requests that GitHub's Copilot coding agent wrote for SigmaHQ's pySigma libraries, which turn Sigma detection rules into Splunk, Elasticsearch and CrowdStrike queries. It stayed quiet on 32 of the 33 that changed functions and flagged one: pySigma #415, where Copilot replaced bundled MITRE ATT&CK data with live downloads. All 1,239 tests pass at that commit, because a fixture added in the same pull request mocks the cache for every test. The five functions that download, parse and cache the data, the core of the change, never ran. Solution. Witnessed is a command-line tool with an IBM Bob workflow. It splits a pull request's diff into the functions it changed and runs the project's own test suite and usage scripts under coverage, including the subprocesses they start. Every changed function gets a verdict: used in real runs, only by tests, or never seen running. For the functions never seen running, IBM Bob, in a custom Witness mode, starts one subagent per function, in parallel. Each writes the smallest script that calls the real function and checks what it returns, and it only counts if it passes a six-rule gate: it exits cleanly, executes the real function body, modifies no package file, calls the target without patching its module, and asserts on the return value with an assert that runs. Target users. Maintainers and platform teams who let coding agents open pull requests and decide what may merge. How they interact. On the pull request branch they run witnessed scan, which prints the gaps and writes a table ready to post on the pull request. They open IBM Bob in the Witness mode to hunt witnesses for the gaps; the mode can only edit files in witnesses/, and every verdict comes from witnessed gate. witnessed report writes a one-page HTML report with the before and after. What makes it unique. It judges each changed function body, separates real use from tests, and refuses to take the agent's word: a witness that swaps the function for a stand-in passes its own asserts and is still rejected, because the real body never ran. Effective and efficient. The signal is rare, so reviewers can trust it: one alert in 33 agent pull requests, and 2 of 88 commits replayed from python-tabulate, both in one pull request whose maintainer later added the missing tests. On that commit the scan took 6.5 seconds to find 2 of 6 functions never run; Bob's subagents wrote a witness for each, fixed them after the gate rejected their first attempts, and both passed, for 1.60 Bobcoins. These numbers count alarms; we did not measure what it misses. Live report: https://kasbsquall.github.io/witnessed/ Code: https://github.com/kasbsquall/witnessed