Intel Bimanual Robot Voice Control MVP

This project implements a full voice-to-robot pipeline for bimanual manipulation in simulation, built for the Intel Physical AI Online Challenge. ARCHITECTURE 1. Voice Input — A browser frontend captures microphone audio and streams raw 16kHz PCM over a WebSocket to a FastAPI backend. The backend proxies it to Speechmatics real-time STT, returning transcripts with low latency using the standard language model. 2. Language Understanding — Transcripts go to a local Ollama LLM (phi3:mini) running on-device. A structured prompt maps natural language to one of five task IDs: pick_place, drawer_open, drawer_close, door_open, or button_press. A keyword fallback handles cases where the model is unavailable. 3. Simulation — Tasks run in MetaWorld (MuJoCo-backed), a standard robotic manipulation benchmark. Each environment has a 4-DOF action space and a 480x480 RGB camera. A live OpenCV window shows the robot camera view as the task executes. 4. Intel Optimization — A ResNet18 vision encoder is exported to OpenVINO IR and benchmarked on Intel hardware. CPU: 10.1 FPS (98.9ms). Intel iGPU: 15.2 FPS (65.7ms). That is a 1.5x speedup from OpenVINO hardware-aware inference. RESULTS 100% task success across all five tasks, evaluated over 10 random seeds each. Average episode: 52 steps, 3.3 seconds. Voice-to-motion latency under 1 second. SUPPORTED TASKS pick_place, drawer_open, drawer_close, door_open, button_press INTEL TECHNOLOGIES OpenVINO converts and runs the vision encoder on Intel CPU and iGPU. The pipeline targets Intel Core Ultra Series 2/3 hardware with NPU support in the benchmark runner. All inference is local — no cloud GPU required.