VoiceSort: language-conditioned bimanual SO-101

Created by team MMHV on September 15, 2026
Intel Online

We built the Intel Online track challenge end to end: a custom MuJoCo scene with two SO-101 arms (public Apache-2.0 assets, procedural objects so there is zero licensing risk), a scripted-demonstration pipeline that writes LeRobot-format data with the natural-language task string kept per episode, and a vision-language-action policy that runs the robot. The "L" is real: each instruction is embedded once with a frozen MiniLM encoder and fed to ACT as a transformer token. Because the conditioning is semantic, the policy follows rephrased commands, not just the three training strings. An instruction-swap rollout shows the language token selecting which arm moves and which object it goes for. Deployment is the required part and it is measured, not claimed: the policy is exported through ONNX to OpenVINO IR with normalization baked into the graph, then executed closed-loop on the i9-13900K CPU (28.9 Hz policy rate) and on the UHD 770 iGPU (28.6 Hz). OpenVINO on the CPU alone is 3.2x faster than PyTorch on the same chip. Success rate on held-out seeds: 13/24 with the FP32 IR, versus 12/24 for the PyTorch reference. A voice layer (Speechmatics STT -> keyword planner -> the same semantic embedding) drives the whole thing from a spoken command; the video shows a recorded instruction going end to end on OpenVINO/CPU.

Category tags: