MISE: Multimodal Bimanual Physical AI

We built MISE to explore a simple question: how can a robot take a natural-language instruction, understand what is happening in the scene, coordinate two arms, and still recover when something does not go exactly as planned? MISE, or Multi-modal Instruction to Skill Execution, is a closed-loop Physical AI system for bimanual manipulation. In our demonstration, the robot receives the instruction “Set the table” and completes a multi-stage task that includes opening a drawer, placing a mug and plate, retrieving a spoon, physically handing the spoon from one arm to the other, and placing the remaining utensils. Instead of asking a learned model to directly control every robot joint, MISE separates reasoning from physical execution. A task-specific vision-language model uses the camera observation and language instruction to determine task progress and the appropriate skill. A validated dependency graph then constrains which action can legally happen next, while camera-guided motor skills perform the physical manipulation. The system continuously verifies task outcomes. If an action does not complete correctly, MISE can detect the failure and select an appropriate validated recovery strategy rather than simply continuing with a bad state. Across our final fixed-seed evaluation, the VLA-guided system completed 10 out of 10 runs, with recovery being activated in 8 of those 10 successful runs. We also exported the learned vision-language component to Intel OpenVINO and ran it on an Intel Core Ultra 7 155H system across the CPU, Intel Arc integrated GPU, and Intel AI Boost NPU. Our goal with MISE is not just to show two robot arms moving. We wanted to demonstrate the complete loop from human instruction and visual understanding to constrained execution, physical verification, recovery, and efficient edge AI deployment.