TraceRCA — Evidence-Grounded AI Incident RCA

TraceRCA is an AI-native production incident investigation and verified remediation platform built with IBM Bob 2.0. Modern AI applications depend on multiple layers including application services, model providers, retries and fallbacks, infrastructure, deployments, and observability systems. When something becomes slow or fails in production, engineers often have to manually correlate telemetry, metrics, configuration changes, provider behavior, and deployment information before understanding what actually happened. Simply pasting logs into an LLM does not fully solve this problem. The model only sees the information manually provided to it, and an AI-generated fix is still only a hypothesis until it is tested. TraceRCA creates a structured evidence layer between production systems and IBM Bob. Operational information is normalized into events, metrics, and changes, together with provenance and correlation metadata. IBM Bob accesses this evidence through our MCP server and dedicated Bob skills, allowing it to investigate incidents using actual system evidence rather than guessing from a single error message. In our working demo, Provider A begins returning HTTP 429 responses. The application retries three times before falling back to Provider B, causing increased latency. TraceRCA captures the complete incident timeline. IBM Bob identifies the repeated 429 responses as the trigger, the retry policy as the amplifier, Provider B as the recovery path, and increased latency as the observed impact. Bob then proposes reducing the maximum retries from three to one. But TraceRCA does not blindly trust the AI recommendation. It sends both the baseline and candidate configurations through a controlled replay engine. A deterministic verifier checks that the candidate still succeeds, preserves fallback to Provider B, reduces retries, and improves latency. Only when these conditions pass does TraceRCA mark the remediation as VERIFIED. Beyond the demo scenario, TraceRCA includes a generic evidence model and a Prometheus connector that converts external observability measurements into normalized evidence that Bob can query using the same interface. TraceRCA is designed for SRE, DevOps, platform engineering, and AI engineering teams that want faster incident investigation while keeping AI reasoning evidence-grounded and remediation verification deterministic. Our principle is simple: don’t ask AI to guess the root cause—give it evidence, and verify the fix.