Voice-Controlled Robot Arm

Developed for the AssemblyAI Voice Agent Hackathon, the Voice-Controlled Robot Arm is a hands-free instrument assistant designed for sterile or hands-occupied settings like operating rooms, where manually searching instrument trays wastes time and risks contamination. The system operates across five core stages: a speech stream utilizing AssemblyAI live transcription, Natural Language Understanding (NLU) for item and correction detection, a Finite State Machine (FSM) to track arm state, closed-loop robot control, and physical simulation in CoppeliaSim. Key technical highlights include closed-loop control that continuously monitors actual joint angles rather than executing blind commands, fuzzy matching (edit-distance matching) to handle imperfect speech-to-text transcriptions, and self-correction logic that allows the robotic arm to abandon a held item mid-task, return it home, and fetch a newly requested item without resetting the system.