Lauren Chagaris@lauren678

Butler AI is a bimanual Physical AI system designed to turn natural-language instructions into coordinated robotic actions for everyday manipulation tasks. The system targets a dinner-table setting in a MuJoCo simulation using two SO-101 robotic arms. The pipeline begins with a voice or text instruction describing a multi-step task, such as setting a table with plates and mugs and pouring water. Butler AI structures the instruction into actionable steps, identifies objects and their positions, selects an appropriate bimanual control strategy, and executes the sequence in simulation. For perception, the system uses ArUco markers and homography-based mapping to estimate object locations relative to the tabletop coordinate frame. The bimanual controller coordinates both SO-101 arms for actions such as picking, holding, pouring, and placing objects. Verification checks are used to determine whether intended actions were completed and to support recovery or retry behavior. We also evaluate robustness using randomized robot-demo trials and investigate Intel-oriented inference optimization with OpenVINO. Our current evaluation includes 10 successful trials out of 10 randomized robot-demo trials. An Intel Core Ultra 7 270K Plus ACT smoke benchmark measured 30.5 ms per 100-action chunk on the CPU, while FP16 NPU inference measured 571.7 ms per chunk. Butler AI provides a reproducible simulation-first foundation for exploring multimodal reasoning, bimanual manipulation, verification, robustness, and edge AI deployment for physical robotics.

🌎 Hybrid Hackathon | 💻 Online Build: September 10–16, 2026 | 📍 On-site: September 15–17, 2026 The official hackathon of AI Infra Summit — co-organized by Kisaco Research and lablab.ai. Build production-grade AI applications on the modern AI infrastructure stack, alongside 8,000+ attendees, researchers, founders, and infra leaders. 📍 On-site Venue (September 15–17): Santa Clara Convention Center 5001 Great America Pkwy, Santa Clara, CA 📍 On-site participation is by invitation only. Travel and accommodation expenses will not be covered.