Totoro Tan@MrPapaya

From spoken intent to executable goals wepth-voice-agent helps users act through speech while making the system’s interpretation and execution path visible. Its central idea is to separate what the user wants from how the system achieves it. Rather than sending a transcript directly into execution, the agent first constructs an explicit, validated SemanticGoal. How it works AssemblyAI provides speech-to-text. Global Jev selects the capability, and Local Jev selects the goal schema. A Llama 3.1 8B structured extractor fills the schema’s arguments, while GoalBuilder validates the proposed values. The deterministic planner then uses the goal and current WorldState to select the required operators. Execution updates that state so the outcome can be inspected. This creates a shared control architecture for different tasks. “Open GitHub” uses deterministic navigation. “Find the Browser Use repository on GitHub” selects a grounded Level 2 browser operator. “Search GitHub for browser-use” instead asks the browser to stop at the site’s search-results page. These requests have distinct goals even though they concern the same website. Try the demo The public Gradio demo accepts recorded speech or typed text and displays the transcript, constructed goal, selected planner operator, execution status, and final URL. Browser actions run in a server-side headless browser. The local runtime additionally supports streaming microphone input, spoken feedback, and desktop capabilities such as Spotify control and native browser actions. The project focuses on structured, observable action execution rather than unrestricted autonomy. Deterministic operators handle supported straightforward commands, while grounded browser capabilities handle more open-ended tasks. Browser.FIND is an extension of that architecture, not the entire product. The prototype remains bounded: website changes, ambiguous results, and insufficient completion evidence can still produce uncertain outcomes.

🌎 Online Hackathon | 💻 Month-long challenge | Dates: Sep 1–30, 2026 | 🏆 $10,000 Prize Pool ($5k cash + $5k in AAI credits) A month-long online challenge run by lablab.ai together with AssemblyAI, the Voice AI infrastructure company for builders. Every participant builds on AssemblyAI. You can join at any point — registration stays open for the whole build window, so you can start on day one or pick the project up halfway through and still submit. 🌍 Fully online hackathon. Join and build from anywhere in the world.