🤝 Top Collaborators
🤓 Latest Submissions
PromptGold
The problem: teams change an LLM system prompt and ship it on vibes. Nobody writes tests for prompts, so leaked secrets, jailbreaks and silent behavior changes reach production. Doing it by hand is slow: read the policy, write a test per rule, paste dozens of attack strings one by one, judge every reply, fix the prompt, repeat. Our solution turns IBM Bob into a prompt QA engineer. promptgold (our open-source pytest plugin for prompts, v0.3 before the hackathon) is the engine; during the hackathon we built the Bob layer on top of it: 1. A "Prompt QA" custom mode with restricted file access, plus workspace rules so Bob follows the repo's testing rules in every mode, on Mac and Windows. 2. A promptgold-init skill: Bob reads a system prompt and its policy document and turns every "must / never / limit" rule into a promptgold test, preferring exact checks over LLM judging. On our sample support bot it produced 9 tests from the policy document. 3. A prompt-scan skill: Bob spawns 4 subagents in parallel (jailbreaks, prompt injections, leak probes, topic escapes — 40 attacks), merges the results, heals the prompt with a separate model, re-scans, and stops before accepting a new baseline. 4. A PreToolUse hook that blocks Bob from running --bless or --no-cassette, so accepting a new baseline or spending money on live calls always stays a human decision. Real result: on our bouncer bot "Velvet", tested with glm-5.3-flash, judged by deepseek-v4.1-flash and healed by gemini-3.8-flash-high (three model families: one is tested, one judges, one fixes), the original prompt failed 4 of 40 attacks; after Bob's heal, 1 remained. A human then blessed 39 golden baselines. All 160 model responses are recorded as cassettes, so the whole scan replays offline for free in CI. Impact: [manual time] by hand vs [Bob time] with Bob for the same scan and fix.
27 Sep 2026
