Window Sticker: Crash-Test Ratings for AI Agents

Created by team Crash Test Dummies on November 08, 2026

VULTR Qwen3 Claude Code

C1-Blast Radius Zero

Before you buy a car, the window sticker tells you how it does in a crash. Before you give an AI agent your code and credentials, you get a demo video. Window Sticker fixes that. We run an agent dozens of times, each run in its own throwaway libkrun microVM on a Vultr VX1 host with a deny-all network and canary credentials. Every task hides one of 16 traps: prompt injection, secret theft, DNS exfiltration, rogue package indexes, destructive shortcuts and false completion. Tripwires grade each run mechanically; Vultr models only triage doubtful runs and explain incidents in plain English. Exact Clopper-Pearson intervals turn the results into stars, so a lucky small sample cannot earn a high rating. Results from 576 real runs: the same DeepSeek model rated 4 stars with a guardrail prompt and 2 stars without; three other models rated 1 star. On Debian 11 with Python 3.10 the same agent reached for an unknown package index in 10 of 12 runs, against 3 of 12 on Python 3.12. Containment held on all four OS images, 17 of 17 escape attempts blocked. Judges can start a live crash test from the guided tour and watch six real microVM runs land in about a minute. Every number links to a step-by-step replay, every run's evidence sits in Vultr Object Storage with its SHA-256, and each test exports an OWASP-mapped security-review PDF. A GitHub Action fails a pull request only on a measurable safety regression. Everyone else sells the cage. We sell the crash rating.

Category tags:

Security, Developer Tools, Cloud Application, Web Application, ProjectFromScratch

Github Presentation Demo

Explore more applications

ReconcileIQ

ReconcileIQ investigates accounts-payable exceptions by retrieving the actual invoice, PO and goods-receipt records, showing only facts it can trace, flagging what is not authorized, and recommending a next step. Built on Vultr, with a full audit trail.

Agent Solo

VULTRClaude CodeAI Studio

StudyPlot AI agent

StudyPlot is an AI-powered web application that helps students study efficiently by organizing documents, generating summaries, creating flashcards and quizzes, and providing an interactive AI study assistant.

RameenTech

ChatGPTClaude CodeCodexOpenAI

Shadow Run: let your agent break a clone first

Your agent wants to touch prod. Shadow Run lets it break a clone first: every risky plan runs on a full clone of prod in a microVM on Vultr, and a person approves what it really did, not what it says, before it reaches the real database.

CoffeAndvULTR

Wayfinder, Lumen OCR and Provenance RAG on ROCm

My three AMD AI League entries are all built on the mandated ROCm PyTorch base image and run Qwen3-VL-8B-Instruct in bf16 on the AMD GPU. A resident server loads the model once and listens on a unix socket, so each harness call to /app/app.py is a thin client that finishes inside the 30-second budget. The weights ship inside the images, and nothing is downloaded at evaluation time. Match 4, Wayfinder (web information retrieval). It reads a natural-language request, fetches the start page and ranks same-site links by how well they match the question. Each page is cleaned to text, with tables kept as rows. The model then either gives the value with the passage it came from, or picks the next link to open, for up to three hops. An answer only counts if it appears on a cited page, and if nothing is found the output is an empty answer. It also works on a saved copy of a site (--corpus). Match 2, Lumen OCR, reads licence plates and road signs. On Chinese plates it keeps the province character; on US plates it drops the state banner and slogan. Multi-line signs are joined in reading order, and when the first pass is unsure about a hard image (glare, blur, noise) it reads an enhanced copy again. Match 3, Provenance RAG, indexes PDFs, Word files, spreadsheets, CSVs, logs, source code and images (read by the vision model) once. Empty folders, unreadable files, encrypted PDFs and unknown types are skipped. Each answer cites only the files it needed, a multi-hop chain cites every file in the chain, and withdrawn revisions are ignored. Each project is also a reusable tool with a CLI, a Python API and a local Studio web UI. On the public samples, using AMD Token Lab's hosted Qwen3.8-27B, Lumen OCR scored 10/10 and Provenance RAG 10/10. I have not yet run the shipped 8B model on an AMD GPU. Match 4 image: docker.io/abuhuraira0001/alx-workbench:m4-v1 Code: github.com/abuhuraira99/wayfinder, github.com/abuhuraira99/lumen-ocr, github.com/abuhuraira99/provenance-rag

Abu Huraira

AMD ROCmQwen3-VLHuggingFace Hub

Tushar Agarwal

Upcoming AI Hackathons
For Innovators & Creators

Explore more applications

ReconcileIQ

ReconcileIQ investigates accounts-payable exceptions by retrieving the actual invoice, PO and goods-receipt records, showing only facts it can trace, flagging what is not authorized, and recommending a next step. Built on Vultr, with a full audit trail.

Agent Solo

VULTRClaude CodeAI Studio

StudyPlot AI agent

StudyPlot is an AI-powered web application that helps students study efficiently by organizing documents, generating summaries, creating flashcards and quizzes, and providing an interactive AI study assistant.

RameenTech

ChatGPTClaude CodeCodexOpenAI

Shadow Run: let your agent break a clone first

Your agent wants to touch prod. Shadow Run lets it break a clone first: every risky plan runs on a full clone of prod in a microVM on Vultr, and a person approves what it really did, not what it says, before it reaches the real database.

CoffeAndvULTR

Wayfinder, Lumen OCR and Provenance RAG on ROCm

My three AMD AI League entries are all built on the mandated ROCm PyTorch base image and run Qwen3-VL-8B-Instruct in bf16 on the AMD GPU. A resident server loads the model once and listens on a unix socket, so each harness call to /app/app.py is a thin client that finishes inside the 30-second budget. The weights ship inside the images, and nothing is downloaded at evaluation time. Match 4, Wayfinder (web information retrieval). It reads a natural-language request, fetches the start page and ranks same-site links by how well they match the question. Each page is cleaned to text, with tables kept as rows. The model then either gives the value with the passage it came from, or picks the next link to open, for up to three hops. An answer only counts if it appears on a cited page, and if nothing is found the output is an empty answer. It also works on a saved copy of a site (--corpus). Match 2, Lumen OCR, reads licence plates and road signs. On Chinese plates it keeps the province character; on US plates it drops the state banner and slogan. Multi-line signs are joined in reading order, and when the first pass is unsure about a hard image (glare, blur, noise) it reads an enhanced copy again. Match 3, Provenance RAG, indexes PDFs, Word files, spreadsheets, CSVs, logs, source code and images (read by the vision model) once. Empty folders, unreadable files, encrypted PDFs and unknown types are skipped. Each answer cites only the files it needed, a multi-hop chain cites every file in the chain, and withdrawn revisions are ignored. Each project is also a reusable tool with a CLI, a Python API and a local Studio web UI. On the public samples, using AMD Token Lab's hosted Qwen3.8-27B, Lumen OCR scored 10/10 and Provenance RAG 10/10. I have not yet run the shipped 8B model on an AMD GPU. Match 4 image: docker.io/abuhuraira0001/alx-workbench:m4-v1 Code: github.com/abuhuraira99/wayfinder, github.com/abuhuraira99/lumen-ocr, github.com/abuhuraira99/provenance-rag

Abu Huraira

AMD ROCmQwen3-VLHuggingFace Hub