🤝 Top Collaborators
🤓 Latest Submissions
Kosha
Problem IBM Bob runs release and deployment work as a fleet: one agent bumps a version, another edits CI, a third runs a migration and deploys. Bob's safety controls judge each of those tool calls alone. That leaves exactly two options today — approve every call manually, which is slow and defeats the point of running agents in parallel, or auto-approve execution, which is fast but blind to what the fleet does as a whole. Nothing sees the bundle. This isn't a hypothetical gap: Per-action limits collapse at fleet scale. Agents individually capped under a shared budget still overdrew it 2.4× at 50 agents, 9.7× at 200, 48× at 1,000 — only a cumulative ledger held every run inside the limit [1]. Agents don't self-limit. Across 2,208 real DevOps prompts, 55.8–67.8% of acting runs crossed an action boundary anyway, and telling the agent the blast radius was high barely moved that number (65.5% vs 64.0%) [3]. The failures that reach production are severe. Of 16,586 real coding-agent GitHub issues, 547 were confirmed safety failures — 326 high or critical, including destructive operations and authorization bypasses [4]. Binary pass/fail hides exactly this. A defense that reported 0% attack success still leaked cross-scope once; judges that see individual actions score a full escalation chain the same as one harmless step [2]. Solution Kosha is a runtime gateway that sits beneath Bob and prices what its fleet actually does, together. Every tool call is classified by reversibility, scope and privilege [2] into a level (L0–L5) and a price, charged against one budget shared by every agent and subagent in the session. Bob keeps auto-approve on. Kosha decides allow, ask — showing a human everything the fleet already did this window — or deny, with a reason and what would be allowed instead. No model makes that call; it's deterministic code, covered by 981 tests. The impact, measured, not claimed: We cut our own false-alarm rate by ~92% through calibration. Early coverage gaps in the effects table meant unmatched commands defaulted conservatively and triggered escalation — 95.2 false asks per 100 benign sessions. Closing those gaps against real SWE-smith and StepShield data brought that to 7.64 per 100, without loosening the rule itself. ~25× fewer interruptions than a per-action gate. At run time, Kosha asks 0.63 times per 100 benign sessions; a gate checking each action alone at L4+ asks 15.53 times per 100, for a lower single-agent catch rate (29.2% vs 58.9%) — the trade Kosha is designed to make. It catches what no per-action check can see, ever. Across 1,800 composed fleets where no single agent's action would trip any existing gate alone, Kosha asks 100% of the time. Denylist and per-action baselines: 0%. Tuned to real work, not guessed. The per-agent cap (375) sits at the 95th percentile of 26,076 real benign sessions. Stated honestly: Kosha also asks on 100% of all-clean composed fleets — it guarantees review after enough consequential work accumulates, it doesn't infer intent. It's a guardrail for well-meaning agents, not an adversary sandbox. References: [1] Irreversibility Budget, arXiv:2609.00275 · [2] Action-Graded Severity Scale, arXiv:2607.07474 · [3] UnderSpecBench, arXiv:2607.02294 · [4] What Breaks When LLMs Code?, arXiv:2605.30777
27 Sep 2026
