🤝 Top Collaborators
🤓 Latest Submissions
DevPulse: grading IBM Bob as a security auditor
Every AI code review demo we've watched goes the same way. The tool lists some problems, everyone's impressed, and nobody asks what it missed or how many of those problems were real. But those are exactly the things a team needs to know before they let an AI reviewer anywhere near their releases. DevPulse tests IBM Bob like an exam. We wrote two small Express apps and hid security bugs in them: 10 textbook ones in the first (SQL injection, MD5 passwords, a hardcoded secret), and 12 subtler ones in the second, an invoicing API (an IDOR, an admin check using jwt.decode instead of jwt.verify, SSRF, path traversal, command injection, prototype pollution and more). The second app also has 7 traps, bits of code that look dangerous but are actually safe. The answer keys stay with us. Bob never sees them. Bob does the real work. It runs in a custom Security Auditor mode that can read the whole project but can only write to the audit folder, so it can't touch the code it's checking. A skill tells it what to look for, Bob splits the files across subagents running in parallel, and writes its findings to JSON. A second skill hands each finding to its own subagent to write a patch. A small script then marks everything: recall, precision, how many traps Bob fell for, and whether each patch actually applies with git apply --check. Bob found all 22 bugs, raised no false alarms and left all 7 traps alone. The patches were the interesting part. In round one Bob picked the right fix every time, but 8 of 10 diffs failed to apply because the line numbers were off. We added one rule to the fix skill, that a patch doesn't count until it passes git apply --check, and in round two all 12 patches applied first time. Who it's for: engineering and security teams already using AI to write or review code, who now have to decide how much to trust it. They plant bugs and traps in their own code, run the Bob skills, and get a real hit rate before an AI reviewer is allowed to block a release. Findings export as SARIF, so they land in GitHub code scanning, and the website lets anyone click through each finding, the exact line Bob flagged and the patch it wrote. What makes it different is that we don't just show Bob's output, we grade it. Most AI review tools are judged on how impressive their findings look. DevPulse measures misses, false alarms, traps and whether the fixes are usable, and it caught a real problem (patches that looked right but wouldn't apply) that a normal demo would have hidden. Every Bob task is recorded in the bob_sessions folder with what it cost, including the one that didn't work, and the whole project used about 7 of our 40 Bobcoins. The site only displays files that Bob and the scoring script wrote.
27 Sep 2026
