BlindSpot

Problem Statement: Development teams are handing more and more real work to AI coding agents like IBM Bob. Before they do, nobody can answer a basic question. How well does the agent actually understand this particular codebase? AI agents rarely say "I don't know". They answer with confidence, and when they are wrong they are often wrong with high confidence. That is the dangerous case, because a confidently wrong belief about the code turns into a confidently wrong code change that a reviewer may not catch. The usual fix is a context file such as AGENTS.md, which the agent reads before every task. Bob can even generate one with its /init command. But no one measures whether these files help, and recent research suggests repository context files often help far less than teams assume. So teams are working blind twice. They do not know where the agent misunderstands their code, and they do not know whether their attempts to fix it worked. Solution Statement: Blindspot measures an AI agent's understanding of a codebase instead of guessing, and uses Bob to improve Bob. First, Blindspot reads the code's syntax tree and generates hundreds of exam questions whose answers are checked automatically against the code. It asks whether classes have certain methods, what default values parameters take, which exceptions functions raise, what functions call and what files import. Some questions ask about names that sound real but do not exist, to catch the agent inventing things. Second, Bob sits the exam closed book through headless Bob Shell, with every tool disabled, and states how confident it is in each answer. Blindspot marks every answer and measures, for each file and function, how often Bob is right, whether its confidence can be trusted, and where it is confidently wrong. Third, the results appear inside Bob IDE through our Bob Readiness plugin. The file tree is coloured by how well Bob understands each file, and functions where Bob was sure but wrong are highlighted, with hover cards showing what Bob believed and what is actually true. Developers can approve findings and have Bob fix them on a separate branch, while areas Bob does not understand are marked as needing a person. Fourth, a custom Bob mode called the Cartographer reads only the modules where Bob made mistakes and writes short, verified notes aimed at exactly those gaps. One command publishes these findings into AGENTS.md and Bob's rules, so every agent working in the repository benefits. Finally, Blindspot proves whether the fix worked. We pre-registered the experiment before seeing any results and tested Bob on a sealed set of 200 questions it had never seen. With the AGENTS.md from Bob's own /init, Bob scored 89 percent. With Blindspot's targeted notes added, it scored 95 percent, fixing 17 answers and breaking 5 (exact McNemar test, p = 0.017). Mistakes fell by more than half, all for a tiny fraction of our Bobcoin budget.