Aashir Noman@azac96547075
🤝 Top Collaborators
🤓 Latest Submissions
Polygraph: Bob can't ship it until it's proven
**Problem.** AI coding agents now write a large share of the code developers ship, and every one of them ends a task the same way: "Done ✅". Developers can't tell which of those claims are true. 66% of developers say AI solutions are "almost right, but not quite" — their top frustration (Stack Overflow 2025, 49,000+ developers). METR found that roughly half of SWE-bench-passing patches would not be merged by real maintainers. A September 2026 paper found agents gaming tests in half or more of rollouts for three open models: deleting assertions, adding skips, changing expected values. A green checkmark no longer means the code is correct, and human reviewers can't re-check everything. **Solution.** Polygraph is a verification gate built into IBM Bob with lifecycle hooks. Other tools guard what an agent is allowed to do. Polygraph guards what it is allowed to claim. - A tamper-evident recorder (PostToolUse hook) logs every action Bob takes into a hash-chained trace stored outside the workspace, so edits to the log are detectable. - When Bob finishes (Stop hook), the verdict engine checks the claim against independent evidence. It re-runs the ORIGINAL tests from the baseline tag against the current code, so edited tests can't fake a pass. It diffs the test files to catch weakened tests (deleted asserts, skip/xfail, changed expected values or tolerances). It checks behaviour: was anything tested after the last edit, and was the full suite run? And it marks each sentence of Bob's claim SUPPORTED or NOT SUPPORTED. - The verdict is VERIFIED, NEEDS REVIEW or REJECTED, with an evidence card explaining why. - If Bob tries to git commit or push an unproven change, the commit gate (PreToolUse hook) blocks it with exit code 2. - The feedback hook (UserPromptSubmit) tells Bob exactly which evidence is missing, so it can fix the work properly. - A "Polygraph Auditor" custom mode (read and run only, no edit rights) launches a blind-verifier subagent that sees only the original ticket and writes its own tests. - A dashboard shows the review queue, evidence cards, session timelines and experiment results. **Impact.** We ran the same tickets through Bob with the gate off and on. Bob was honest in all 3 recorded runs — every claim sentence checked out against the evidence — and Polygraph proved that automatically in about 3 seconds per task instead of by hand. It still rejected one of those honest runs: Bob's ticket-02 fix was correct, but the independent run of the original test suite came back 24 passed / 4 failed, so "done" was not shippable. When we weakened a test by hand (one assert deleted from test_cart.py), Polygraph flagged `deleted_asserts`, returned REJECTED, marked exactly the two success-claiming sentences NOT SUPPORTED, and blocked 5 commit attempts — 0 unproven commits landed. Across the recorded gate runs: 10 commit attempts blocked, 0 unproven commits. The end-to-end self-test passes 5/5, including tamper detection — an intact trace verifies VALID, and a trace with a single altered entry is caught by a hash mismatch, so the agent cannot rewrite its own history. **How Bob powers it.** Bob built Polygraph, and Polygraph runs inside Bob. It uses 4 lifecycle hooks (PostToolUse, Stop, PreToolUse, UserPromptSubmit), AGENTS.md rules, a custom mode, a subagent and a Skill, and Agent mode throughout the build. Every task-session summary is in `bob_sessions/`.
27 Sep 2026
OPTIONS ALPHA
A broker API LLM is a liability. Can create a premium, or position size wrong, or open a spread whose legs fill separately, leaving a naked short. Most agents mistake authority for judgment. We keep them apart. 14 LLM Agents Research a Job It trades or not, according to deterministic code. 14 agents in 4 teams run a LangGraph state machine: five parallel analysts, a bull/bear debate with a manager who adjudicates, a trader and options strategist, then a three-stance risk debate whose judge can overrule the trader. “Agents choose the shape of a trade. They never price a trade. A deterministic gate re-prices every leg from live bid/ask, re-computes net premium and max loss, and vetoes invalid quotes, spread > 20% of mid, risk-sized loss > 2% of equity, debit above buying power, collateral shortfall, or any unbounded loss. It re-runs on new quotes before submission and drops the plan if the judge changes direction. So says a regression test: The model requires a $99.99 premium on a $2.00 spread-the gate sends $2.00. Then you manage positions. Legs group into structures (a 4-leg condor is one position) and close on a stop (1.5x credit, or 50% of debit), a target (35% of premium) or 21 DTE - as a single MLEG order, on a timer independent of the agent graph. Powered by Alpaca paper trading on the official Alpaca MCP server (uvx alpaca-mcp-server) with alpaca-py as fallback. Uses MLEG spreads, bracket/OTO equity stops held at the broker, and portfolio history for the equity curve. Now, a web app with seven workspaces includes a live dashboard, an Options Desk with each veto reason, and a ledger that links each order to the run that created it. 433 offline tests Couple filled trades, honest limit. We demonstrate governed independence, limited power, predictable hazard, complete auditability, not a verified advantage. Aashir Noman—AI Engineer Almas Sultan—Economist
4 Sep 2026
