EcoRoute: 3-Tier Agentic Router

Standard AI application wrappers burn expensive cloud tokens and introduce unnecessary latency by sending every simple, generic user prompt to massive 70B parameter models. EcoRoute solves this unit economics problem by introducing a 3-Tier Agentic Triage Engine. By deploying a lightweight, local 0.5B AI (Qwen 2.5) directly within a multithreaded Docker container alongside a deterministic heuristic gatekeeper, EcoRoute evaluates task complexity in real-time. Tier 1 (Trivial): The local model intercepts and answers simple facts, grammar corrections, and summaries for absolutely zero cloud tokens. Tier 2 (Medium): Tasks escalated by the local model due to uncertainty are routed to a highly efficient, lower-cost cloud model (e.g., Llama 3 8B). Tier 3 (Complex): The deterministic Python gatekeeper scans for heavy coding or mathematical logic, bypassing the local model entirely to route directly to a premium model (e.g., Llama 3 70B). Beyond routing logic, EcoRoute is built as an indestructible, enterprise-grade backend. It utilizes tenacity for Exponential Backoff to gracefully survive API rate limits (429 errors). It features dynamic context string truncation to prevent local memory overflow (OOM) on massive prompts. Finally it uses thread-locked incremental file saving to guarantee strict JSON schema validation and ensure partial credit is preserved even if a grading server enforces strict execution timeouts. EcoRoute maximizes the API budget without sacrificing a single percentage point of accuracy.