
AI agents trust everything they read, so hidden instructions inside a file, web page or tool output can trick them into leaking secrets. Cars are crash tested before they are sold, but agents have no simple, repeatable test. ChaosPilot fixes that. It drops an agent into a disposable Docker container with a fake canary secret, then attacks it with 15 prompt-injection scenarios, each in its own fresh container and isolated network. A trap listener outside the container watches for the secret. If it reaches the listener or the agent’s final answer, that scenario is marked as a leak. A control scenario with no attack proves the setup works. The result is a clear PASS, FAIL, ERROR or INCONCLUSIVE verdict with a reason for each scenario, and an error is never counted as a pass. The model under test is called through Vultr Serverless Inference, and everything runs for real: real containers, a real model, real results. I left out any claim about running on a Vultr VM, since your README doesn’t say it. If your app is deployed on a Vultr VM, add this sentence at the end: “The Node.js server and the Docker sandboxes run on a Vultr VM backend.”
8 Nov 2026