TableMate — Voice-Driven Bimanual Physical AI

TableMate is a voice-driven Physical AI system for intelligent bimanual robotic manipulation in restaurant and table-service environments. The project uses two simulated SO-101 robotic arms in MuJoCo to transform natural-language instructions into coordinated, multi-step physical actions. Rather than executing a fixed sequence, TableMate represents restaurant requests as structured task plans and dynamically assigns actions to the left arm, right arm, or both arms cooperatively. The system supports parallel operations such as placing a plate and cutlery simultaneously, as well as cooperative bimanual tasks such as transporting a main dish. Current scenarios include fine dining, quick lunch, shared dinner, selective table clearing, and changing guest requests. TableMate is designed around the hackathon’s multimodal Physical AI challenge: combining language understanding, visual scene information, task context, bimanual coordination, manipulation, and verification. The architecture is being developed to support randomized MuJoCo environments and evaluation across multiple seeds to measure robustness to changes in object placement and scene conditions. The deployment pipeline targets Intel Core Ultra Series 2/3 and uses OpenVINO for supported AI inference optimization. The project also incorporates Intel Physical AI tooling and a reproducible simulation, evaluation, and benchmarking workflow. Speechmatics integration extends the interface from written commands to real-time spoken instructions, allowing a guest or operator to naturally request actions such as setting, serving, modifying, or clearing the table. The goal of TableMate is to demonstrate how Physical AI can move beyond isolated robotic actions toward an interactive agent that can understand what a person wants, reason about a changing environment, coordinate two arms, execute safely, and verify that a real-world-style task has been completed correctly.