RescueHandsAI: A dinner table that survives a drop

Created by team Healmezcat on September 16, 2026
Intel Online

*The problem.* A vision-language-action policy always outputs an action, even when reality has already gone wrong. In two-arm manipulation, one slip by one hand ruins the other hand's work. RescueHands AI tests a simple idea: never trust an action just because a model produced it. Check what physically happened, and recover. *The task.* Two SO-101 arms in MuJoCo share a dinner table. The instruction names a fork or a spoon, and both lie on a mat in random order, so the words decide which one to move. The right arm picks up the named utensil, passes it to the left arm **in the air**, the left arm places it beside the plate, and the right arm places the cup. Each seed changes positions, sizes, mass, friction, lighting and table colour. Grasps are real contacts: nothing is welded or teleported. *The learned policy.* SmolVLA sees three cameras (overhead and both wrists), 12 joint positions and the instruction, and outputs 12 joint targets in 50-step chunks. There is no inverse kinematics in the learned control loop. A scripted IK teacher only makes the demonstrations. We fine-tuned on free Kaggle T4 GPUs: (v1: 101 demonstrations.), (v2: 569 demonstrations (336,729 frames). *Safety and recovery.* An action guard enforces joint limits and step size. A physics auditor reads simulator state that the policy never sees: is the item held by both jaws, supported, in its zone, dropped? When an item really drops, the supervisor opens both hands, returns to a safe pose, and lets the policy try again, at most twice. Success is measured from physics, never from the policy's claim. It requires both items stable in their zones, released and upright, the spare utensil still near its start at the end, and an ordered in-air hand-off. Collision checks cover arm-to-arm contact only. Randomized item friction does not reach the jaw contacts. Simulation pauses during inference, so this is not real-time 20 Hz control.

Category tags: