🤝 Top Collaborators
🤓 Latest Submissions
Gaia-Ground Truth For AI Written Code
AI now writes the code, the tests and the docs, and then grades its own homework. Three claims ship unchecked: the code matches the spec, the tests prove it works, and we'll fix it fast when it breaks. Nobody reads a spec against the code clause by clause. Coverage counts lines touched, not bugs caught. And the on-call engineer has to rebuild the story from logs under pressure. THE INSIGHT The checker never sees the code it checks. Gaia's isolation wall is declarative IBM Bob configuration (custom-mode tool groups and fileRegex write fences), not a prompt asking nicely. The Witness mode cannot write to src/. Only the Surgeon mode can. THREE COURTS Witness (Legal): Bob reads the API spec, extracts every MUST clause, writes tests from the spec alone, and runs them against the real implementation through Gaia's MCP engine. The output is a clause wall pinned to file and line. On our sample payments API it covered 8/8 clauses and caught 8 real violations (19 failed assertions) in a single Bob task, for 3.94 Bobcoins. TrustGap (Honest): the Mutineer mode runs Stryker on a scratch copy of the code. The Isolate mode then traces every surviving mutant to the test that should have caught it. Claimed line coverage was 91.7% against an 89.5% honest kill rate (20 mutants, 17 killed). Two tests covering the discount logic killed zero mutants. TrustGap named each one, for 1.05 Bobcoins. Triage (Survivable): paste an incident into the incident-commander mode. Parallel explore subagents read deploys, logs, metrics and the runbook PDF at once. They isolate the suspect deploy, find the root cause (pending holds added to available balance) and draft the postmortem. Signals come from a local MCP fixture server: deterministic, offline, no API keys. Every Surgeon fix is a Bob checkpoint that can be rolled back. HOW IBM BOB POWERS IT • 9 custom modes, each with its own write fence • 11 SKILL.md workflows: extract, test, audit, mutate, isolate, intake, forensics, patch, postmortem • Parallel explore subagents • A model-free MCP court engine over stdio • Document understanding for the spec and runbook PDFs • Checkpoints and rollback Bob task session summaries are in challenge/bob_sessions. ARCHITECTURE The court engine never calls a model. Verdicts come from Jest and Stryker, so the numbers can't be hallucinated. The host's model directs the work. Gaia installs its court subagents into IBM Bob, Claude Code, Codex or VS Code Copilot chat modes. Each repo needs one YAML adapter, and Gaia auto-detects it. THE PRODUCT • VS Code extension with a 3-Court panel (Config, Run, Last report, Dashboard, Install courts), packaged as a .vsix • Live dashboard that streams mutation results in real time • In-loop verify-diff tool that lets any coding agent run its patch through the courts before commit • Reports as interactive HTML and agent-readable Markdown WORKFLOWS IMPROVED Spec compliance review, test review, incident debugging, and guardrails for AI coding agents. WHAT'S NEXT A CI merge gate that blocks on red clauses, IBM watsonx Granite as a host model, and a status-bar verdict that's always visible. Legal. Honest. Survivable. Ship it.
27 Sep 2026
