AI Incident Commander

Running inference on a larger scale is nothing like web operations. For instance, a GPU can thermal-throttle without crashing, the KV cache can fill up and start preempting sequences, and a node can hang on loading weights while still taking traffic, since when load balancers direct traffic towards the nodes that are underutilized, even having the idle node looks like having good points of minimal load out of the complete set of equipment. These are six different root causes of an issue that on the dashboard will show "p99 is going up." Two things that are helpful to look at are a thermal-throttled node and a real traffic spike, which produce virtually the same anomaly. However, by making a different decision in those two cases one can either drain the node or scale it up. When you drain the node during the spike, you'll worsen the situation. The Incident Commander operates over a simulated fleet with a closed-loop setup. Its detection process is deterministic, with no model involved in the process, and makes decisions based on either prolonged SLO violations or regressions against the fleet benchmark. This allows grey failures to be detected by considering a node with a faulty redeployment marked by real-life metrics above 1.7× of a fleet benchmark to be functioning well below SLO. Afterward, the use of LLM involves the selection of a certain telemetry set from five tools that will provide a root cause accompanied by evidence, along with the excluded reasons. A policy layer will now vet the proposed measure and may either approve or reject it. During one of the tests, a throttled node was drained with no substitute present, which resulted in an increase in p99 from 5250ms to 9350ms, so policy rewrites this action to let the fleet save up capacity while draining the node. The verifier then observes the entire process until the fleet returns to normal, tracking the decreasing backlog but still keeping in mind that latency is high.