TestPilot — AI-Assisted Software Testing

Software testing is essential but often repetitive and time-consuming. Developers need to identify untested code, write tests, execute them, diagnose failures, repair tests or code, and repeatedly verify the result. AI can automate parts of this workflow, but blindly allowing an AI agent to modify tests can create a dangerous outcome: the system may change a test simply to make it pass while leaving a genuine application defect hidden. TestPilot is an AI-assisted software testing pipeline designed to automate this workflow while keeping verification measurable and safety-aware. The system analyzes a Python repository to identify testing gaps and target modules. A Test Generation Agent then creates tests using the project's existing testing conventions. Generated tests are extracted and validated through an AST-based validation layer before execution. TestPilot executes the generated tests and analyzes failures. When appropriate, a bounded Repair Agent attempts to repair failing tests through controlled iterations. A Safety Gate distinguishes failures that should not be "fixed" by modifying tests, such as genuine source-code logic defects. The pipeline follows: Analyze → Generate → Validate → Execute → Detect Failure → Analyze Failure → Repair → Re-execute → Evaluate → Metrics Target users are software developers, engineering teams, and researchers interested in AI-assisted software quality and automated testing. Users interact with TestPilot primarily through a CLI. They can provide a repository, select a target, run generation or evaluation, and inspect structured results including generated-test validity, execution outcomes, repair attempts, coverage changes, and benchmark metrics. The creative aspect of TestPilot is not simply generating more tests. It combines autonomous test generation with execution feedback, bounded iterative repair, safety checks, and measurable evaluation. The system is designed so that an AI agent is not rewarded merely for making every test pass. In a real benchmark run, TestPilot generated 8 tests and all 8 were valid. Coverage increased from 65.12% to 69.77%. During repair, the system encountered a genuine logic failure; after two repair attempts it stopped and triggered its safety gate rather than modifying tests to conceal the defect. The final project verification completed with 342 passing tests and mypy reporting no issues across 27 source files.