DeleteProof

The problem A customer asks to be deleted. The API deletes the row and commits. But a profile update for that customer was already waiting in the queue. The worker processes it after the delete, runs an ordinary upsert, and the customer is back. In our recorded run, the deleted row reappeared 18 ms after the delete committed. This breaks GDPR's right to erasure, and it is hard to catch. It only happens when an API transaction and a queue worker interleave in one specific order, so tests pass and the bug ships. Teams find "deleted users coming back" in production and ship a fix nobody can prove. Our solution DeleteProof makes the race reproducible and turns "is it fixed?" into a yes or no answer backed by the database: 1. Reproduce: Redis barriers, not sleeps, force the exact interleaving against real PostgreSQL 17 and Redis 7, so the row comes back every run, not once in a thousand. 2. Explain: API, worker and database events are merged into one timeline showing which transaction wrote after the delete committed. 3. Repair: a reference fix. Delete and worker both take a PostgreSQL advisory lock on the customer, the delete leaves a tombstone, and the worker checks it before writing. 4. Verify: five scenarios run against both the vulnerable and fixed versions. The fix must block stale writes and still apply legitimate updates, because a fix that blocks everything is also wrong. Result: 10 of 10 expected outcomes, plus 6 passing integration tests. Target users Backend and platform engineers running queue workers (BullMQ, Kafka, SQS), SRE teams chasing data that should not exist, and privacy teams who must show deletions stick. How they interact with it - CLI: npm run dp -- verify runs the whole matrix and prints a verdict with a color-coded trace. - Dashboard: run scenarios, compare vulnerable and fixed timelines side by side, and see the final database state. - IBM Bob via MCP: six tools let a developer ask Bob to reproduce the bug, read the traces and explain the fix without leaving the IDE. Every run is saved as a JSON evidence report for a code review or audit. What makes it unique DeleteProof treats a concurrency bug as something you can replay on demand. The verdict comes from the real database state, not mocks or logs. It checks both directions, so a fix cannot pass by breaking the feature. And it is built for an AI assistant: Bob runs it, reads the evidence and reasons about the repair, instead of a developer hand-building a race harness. Why it works A race that takes days to catch now takes one command: 455 ms for our vulnerable run, 127 ms for the fixed one. The repair is a few lines of standard PostgreSQL. A fix passes only when the database shows the deleted customer stays deleted and everyone else keeps working.