Vino & Dine: Voice-Guided Bimanual SmolVLA

Vino & Dine is a voice-guided bimanual robotics project in which two simulated SO-101 arms complete a multi-step table-setting task in MuJoCo. The goal is to connect a spoken request to a vision-language-action policy that can be evaluated safely before hardware deployment. The system uses an overhead camera, two wrist cameras, domain randomization, deterministic evaluation, and custom grasp-assist logic. Demonstrations generated by an IK-based expert are stored in LeRobot format and used to fine-tune SmolVLA with the SmolVLM2-500M-Video-Instruct backbone. The policy outputs 50-step action chunks for the 12-dimensional bimanual joint state. Speechmatics provides cloud-based real-time speech recognition. A spoken command is transcribed and matched to trigger keywords, which then launch the real SmolVLA rollout using the trained drawer-and-plate instruction. Verified results include 10/10 successful drawer open/close episodes and 7/10 successful plate pick-and-place episodes. Fixing an image-preprocessing mismatch improved evaluation from 2/10 to 7/10 without retraining the checkpoint. For optimized inference, we built a SmolVLA-to-OpenVINO bridge that directly exports the fixed-shape velocity step to OpenVINO IR while retaining PyTorch preprocessing and Python-based Euler updates. CPU parity testing validated the implementation, and our benchmark showed about a 20× speedup for the denoising-step workload versus PyTorch CPU execution. The runtime supports OpenVINO CPU, GPU, and NPU targets. The cup extension is still in progress, currently succeeding on 4/10 evaluation seeds, so it is not treated as a headline result. Future work will improve cup reliability and extend the voice-triggered workflow to the full drawer, plate, and cup sequence.