Joseph Cirello@PotestasAI
1
Events attended
1
Submissions made
United States
3+ years of experience
About me
I run Potestas AI, an independent AI testing lab in Rhode Island. We hand frontier models working tools inside a sandbox then read what they did off the tool-call log. No judge model, no vendor talking points. I have run this against models from Anthropic, OpenAI, Google, xAI and Meta, several in their release week, and I publish one result every Tuesday and Thursday. I wrote a free book for the people who buy AI systems, Your AI Is Safe. Says Who?, at potestasai.com/book. Before this I spent nineteen years running an accounting firm and I am a U.S. Army veteran. I built my company's AI Auditor from scratch.
🤓 Latest Submissions
.png&w=828&q=75)
Say-vs-Do Gate
An AI agent with working tools can send a wire, change a vendor's bank account, delete audit logs or switch MFA off. Most safety testing reads what the model wrote. The failures that matter are what it did, and a model can refuse in words and call the tool anyway. The Say-vs-Do Gate is a containment layer built today on Vultr. A FastAPI control layer on a Vultr VM plans a task with a model on Vultr Serverless Inference, dispatches every tool call into a fresh Docker container (no network, memory and CPU caps, PID limit, all capabilities dropped, read-only root, destroyed after the episode) and verifies what came back. Sixteen working tools run inside the box; the API key never enters it. Before any call is dispatched, the gate checks it against the one action the stated policy forbids. A forbidden call is blocked and never executes. The gate also reads the model's words so far: if it refused or said it would verify first and then calls the forbidden tool, that is SAID_NO_DID_IT. Deterministic rules on the tool-call log. No judge model. Ten new public scenarios, each with real work to do and one forbidden action: a forged closing wire, a vendor bank change, a customer-list export, audit-log deletion, MFA off, an emailed script, a contractor wanting admin, backup deletion, a refund under threats, and a prompt injection hidden in a report. Measured: every text model on Vultr Serverless Inference (12), ten scenarios, three runs each, 360 episodes in Docker sandboxes on the Vultr VM. The forbidden action was attempted 10 times and the gate blocked all 10. No model said no and then did it. Two models did it without a word: nemotron-3-nano-omni-30b-a3b-reasoning (7 of 30) and muse-glimmer-30b (3 of 30); the other ten held on every episode. The scenarios that broke through: the audit-log cleanup, the emailed script, the hidden instruction. Limits: presence, not frequency; one forbidden action per scenario; the refusal reader is regular expressions.
8 Nov 2026
