InferFlow

InferFlow helps a small merchant's support operator answer customer questions while background ticket summaries compete for limited inference capacity. AMD Token Factory powers live drafts and summaries with DeepSeek-V4-Flash. The Python/FastAPI gateway combines bounded priority queues, paced requests, 429 backoff, adaptive concurrency, tenant- and knowledge-version-scoped exact caching, cancellation, and a persistent SQLite call ledger. The workbench exposes queue progress, latency, usage, and downloadable evidence. In one sequential comparison of six synthetic requests per arm at 20 RPM and concurrency two, interactive median latency was 13.946s for FIFO without caching versus 8.363s for priority scheduling with caching. Background median rose from 5.826s to 11.831s. Both returned 6/6 HTTP successes; one exact cache hit reduced upstream generations from six to five. This tiny sample combines scheduling and caching effects and proves neither statistical significance nor GPU acceleration. Quality review found both arms incorrectly accepted a day-10 return under a seven-day policy. The current structured precheck uses deterministic rules, asks for missing facts, and routes qualifying cases to human review. No refund executes automatically; free-form drafts remain fallible. A later real UI run completed three summaries and one customer question in five upstream calls including model discovery. Exact inputs, outputs, failures and limits are preserved. Fifty tests cover scheduling, budgets and policy rules, including browser/Python agreement across 45 boundary cases. The public demo provides interactive browser-side return checks and recorded AMD evidence. Live inference runs through the local gateway with a private API key. The English walkthrough uses real UI screenshots, edited pacing and synthetic narration. Source is public under MIT. No merchant pilot or production-readiness claim is made.