SignPulseAI: Real-Time Gesture-to-Text Translator

SignPulseAI is a real-time multimodal AI application designed to help bridge communication gaps for people who may have difficulty hearing. The application combines hand gestures or sign-language input captured through a camera with spoken input captured through a microphone. The system uses MediaPipe Hand Landmarker for real-time hand landmark detection and processes the visual input together with speech information through a low-latency AI pipeline. Gemini Multimodal AI is used to interpret the combined visual and language context and determine the intended meaning. The interpreted result is presented as readable text in real time through a live user interface. Users can see the camera feed, gesture detection, and generated text while interacting with the application. The core workflow is: Camera + Microphone → Hand Gesture & Speech Input → Computer Vision → Gemini Multimodal AI → Interpretation → Real-Time Text Output. SignPulseAI focuses on accessibility, real-time interaction, and multimodal communication. Instead of relying only on visual gesture recognition, the system combines gesture and spoken context to improve interpretation and provide meaningful text output. The project was developed by Team Algorithmic Vanguard for the AI Factory Hackathon 2026 using native.builder as the primary application-building platform, with Gemini Multimodal AI, MediaPipe, Speechmatics, Supabase, React, and Vite as supporting technologies.