Butler AI: Bimanual Physical AI

Butler AI is a bimanual Physical AI system designed to turn natural-language instructions into coordinated robotic actions for everyday manipulation tasks. The system targets a dinner-table setting in a MuJoCo simulation using two SO-101 robotic arms. The pipeline begins with a voice or text instruction describing a multi-step task, such as setting a table with plates and mugs and pouring water. Butler AI structures the instruction into actionable steps, identifies objects and their positions, selects an appropriate bimanual control strategy, and executes the sequence in simulation. For perception, the system uses ArUco markers and homography-based mapping to estimate object locations relative to the tabletop coordinate frame. The bimanual controller coordinates both SO-101 arms for actions such as picking, holding, pouring, and placing objects. Verification checks are used to determine whether intended actions were completed and to support recovery or retry behavior. We also evaluate robustness using randomized robot-demo trials and investigate Intel-oriented inference optimization with OpenVINO. Our current evaluation includes 10 successful trials out of 10 randomized robot-demo trials. An Intel Core Ultra 7 270K Plus ACT smoke benchmark measured 30.5 ms per 100-action chunk on the CPU, while FP16 NPU inference measured 571.7 ms per chunk. Butler AI provides a reproducible simulation-first foundation for exploring multimodal reasoning, bimanual manipulation, verification, robustness, and edge AI deployment for physical robotics.