Real-Time Voice AI Agent

This project is a Real-Time Voice AI Agent designed to provide natural, hands-free interaction between users and artificial intelligence through spoken conversation. The system captures a user's voice through a web browser, converts speech into text using real-time speech recognition, processes the request with a large language model, and converts the generated response back into speech. The architecture uses AssemblyAI for real-time speech-to-text, Gemini for AI response generation, streaming text processing, text-to-speech, and Web Audio/AudioWorklet for browser-based audio playback. Instead of waiting for the complete AI response, the system processes and streams response chunks progressively to reduce perceived latency and improve conversational flow. A major engineering focus is real-time stability and interruption handling. The system uses turn IDs, conversation management, streaming generation, audio queue scheduling, timeout handling, and barge-in support. When a user begins speaking while the assistant is responding, the active LLM generation, TTS processing, and queued audio can be cancelled so the new request becomes the active conversation turn. During development, major issues such as long response delays, incorrect LLM configuration, failed AI connections, duplicate requests, unstable audio playback, and misleading connection states were identified and addressed through systematic debugging, health checks, improved error handling, streaming architecture, and turn-aware audio management. The deployment architecture is designed for Vercel frontend hosting and Render FastAPI backend hosting, while keeping API credentials securely on the server side. PersonaPlex is maintained as an optional future speech-to-speech engine when suitable GPU infrastructure is available.