Speechmatics API

The Speechmatics API is the company's core speech-to-text service, providing batch file transcription and real-time streaming transcription via WebSocket. Powered by the Ursa 2 model (released October 2024), it supports 55+ languages and dialects, speaker diarization, automatic translation into 30+ target languages, and a suite of Voice Intelligence add-ons. Transcription requires no model fine-tuning; custom dictionaries of up to 1,000 words take effect immediately.

General
Release date	Generally available; Ursa 2 model released Oct 2024
Developer	Speechmatics
Type	Cloud speech-to-text API (batch and real-time)
License	Commercial API
Documentation	docs.speechmatics.com/speech-to-text
GitHub	speechmatics/speechmatics-python-sdk

Core Features

55+ languages and dialects: broad multilingual support including accent and dialect variants.
Two accuracy tiers: Enhanced (optimized for accuracy) and Standard (optimized for speed and cost).
Speaker diarization: multi-speaker detection included at no extra cost in all plans.
Custom dictionary: up to 1,000 domain-specific words added without retraining.
Automatic translation: transcripts translated into 30+ target languages via AI.
Voice Intelligence add-ons: summarization, sentiment analysis, topic detection, chapter generation, and entity recognition.
Audio events detection: identifies non-speech events in audio.
Smart formatting: formats numbers, dates, currencies, and capitalization automatically.
Sub-1-second real-time latency: streaming transcription via WebSocket.
Flexible deployment: cloud API, on-premises, on-device, Docker, and Kubernetes.

Accuracy Benchmarks (Ursa 2)

Metric	Result
WER on Kincaid46 (English)	7.88% (surpasses human-level on that test)
WER improvement vs. previous Ursa	18% reduction across 50+ languages
FLEURS dataset leadership	Leads in 62% of supported languages
Head-to-head vs. other providers	Wins 88% of comparisons

Pricing

Tier	Included	Rate
Free	480 minutes/month	No credit card required
Pro	Up to 6,000 hours/month	From $0.24/hour (with discount)
Enterprise	Unlimited scale, no rate limits	Custom

Volume discounts apply automatically above 500 hours per month per transcription type.

Tools and Resources

Batch API Reference: REST API for asynchronous file transcription jobs.
Real-time API Reference: WebSocket API reference for streaming transcription.
Python SDK: official SDK covering STT batch, real-time, and TTS.
JavaScript/TypeScript SDK: official browser and Node.js SDK.
Developer Portal: API key management and usage monitoring.

Ecosystem and Integrations

Integrates with LiveKit, Pipecat, and Vapi for voice pipeline deployments.
Available on Microsoft Azure Marketplace.
Compatible with on-device and edge deployments via Docker or Kubernetes.
Medical Model variant targets clinical transcription in English, German, Danish, and Norwegian.

Start building with the free tier (no credit card required) and explore the full API via docs.speechmatics.com.

Nexus

Nexus is a multi-tenant social messaging platform that unifies Telegram, WhatsApp, and LinkedIn conversations into a single, real-time inbox. Built on Next.js, MongoDB, and Socket.io, it gives support teams one place to read, reply, and manage messages across all channels — with Speechmatics-powered translation so agents can communicate in any language without leaving the thread. Role-based access, a BullMQ task queue, full-text search, canned responses, analytics dashboards, and SLA timers make Nexus a complete customer-communication layer for teams of any size.

TableSync: Dual-Arm Robotic Manipulation

TableSync is a bimanual manipulation framework for coordinated multi-arm desktop manipulation tasks in simulated physics. It enforces a strict, type-safe boundary between high-level multimodal reasoning (Gemini API emitting structured Pydantic plans) and deterministic low-level physical execution (damped least-squares inverse kinematics, waypoint generation, and contact-state verification over MuJoCo physics). TableSync decouples perception and reasoning from real-time joint control through a four-tier architecture: +-------------------------------------------------------------------------+ | Perception & Reasoning Layer | | - Multimodal camera perception (overhead camera RGB frames) | | - Task decomposition & atomic action sequencing | | - JSON-mode structured output validated against Pydantic schema | +------------------------------------+------------------------------------+ | PlannerOutput (JSON) v +-------------------------------------------------------------------------+ | Interface Boundary (`contracts.py`) | | - Enums: ArmID, ActionType, TargetObject, SubtaskStatus, HandoffPhase | | - Models: Subtask, PlannerOutput, ExecutionResult, HandoffState | | - Mutual reach waypoint: HANDOFF_POINT_WORLD = (0.0, 0.15, 0.30) | | - Safety contract: GRIP_DWELL_FRAMES = 10 consecutive contact checks | +------------------------------------+------------------------------------+ | Validated Subtask Sequence

BimanualIQ

BimanualIQ is a voice-controlled Physical AI system designed for intelligent two-arm robotic manipulation. It combines natural-language understanding, visual perception, multimodal reasoning, motion planning, and coordinated execution to let users control two robotic arms through simple spoken instructions. In the core demo, a user can say, “Set the table for two, and do not use the red cup,” and BimanualIQ interprets the request, detects objects in the scene, builds a task plan, assigns actions to each arm, and completes the table-setting workflow inside MuJoCo. The system uses Speechmatics for real-time speech recognition, camera observations for scene understanding, and a VLA-based reasoning layer for translating intent into physical actions. A bimanual coordinator decides when the arms should work independently, perform handoffs, stabilize objects, or execute synchronized movements. BimanualIQ also monitors execution continuously, allowing it to detect failed grasps, misplaced objects, unexpected disturbances, and new voice commands, then replan without restarting the entire task. OpenVINO optimizes perception and AI inference for Intel Core Ultra Series 2/3 hardware, while seeded simulation tests measure task completion, robustness, latency, and generalization across randomized environments. The result is a practical, adaptive, and reproducible platform for conversational bimanual robotics. It demonstrates how spoken intent can become physical action.

KINLOCK: Bimanual VLA Flight Envelope Protection

Vision-Language-Action (VLA) models propose actions at 10–30 Hz, while physical robot servos execute at 1,000 Hz. In bimanual tasks—such as two robot arms carrying an object together—this frequency gap causes kinematic drift between the end-effectors. Documented in Hugging Face LeRobot Issue #3154, this drift generates over 2,000 N of internal tensile and compressive force, stripping servo gears and destroying physical hardware. KINLOCK provides deterministic, sub-millisecond Flight Envelope Protection (FEP) for dual-arm robotic systems. Positioned directly between any high-level VLA model and low-level servo controllers, KINLOCK projects unconstrained velocity proposals onto the null-space of closed-chain kinematic constraints using Moore-Penrose pseudo-inversion and Baumgarte drift stabilization (α = 12.0 s⁻¹). Benchmarked against dual SO-101 6-DOF arms on a 150-frame table-setting dataset, KINLOCK reduces peak internal strain from 2,033.99 N down to 0.02 N (100% attenuation) within a 72.5 μs callback dispatch, strictly preserving the 348.24 mm grasp invariant without interrupting ongoing task execution. Additionally, KINLOCK integrates Speechmatics Realtime WebSocket API for acoustic emergency reflexes. When an operator utters "STOP" or "HALT", KINLOCK parses the phoneme stream and triggers an exponential admittance deceleration curve: bringing arm velocities to a complete halt in under 50 ms while sustaining 100% gripper clamp torque so carried objects never drop.

Intel Bimanual Robot Voice Control MVP

This project implements a full voice-to-robot pipeline for bimanual manipulation in simulation, built for the Intel Physical AI Online Challenge. ARCHITECTURE 1. Voice Input — A browser frontend captures microphone audio and streams raw 16kHz PCM over a WebSocket to a FastAPI backend. The backend proxies it to Speechmatics real-time STT, returning transcripts with low latency using the standard language model. 2. Language Understanding — Transcripts go to a local Ollama LLM (phi3:mini) running on-device. A structured prompt maps natural language to one of five task IDs: pick_place, drawer_open, drawer_close, door_open, or button_press. A keyword fallback handles cases where the model is unavailable. 3. Simulation — Tasks run in MetaWorld (MuJoCo-backed), a standard robotic manipulation benchmark. Each environment has a 4-DOF action space and a 480x480 RGB camera. A live OpenCV window shows the robot camera view as the task executes. 4. Intel Optimization — A ResNet18 vision encoder is exported to OpenVINO IR and benchmarked on Intel hardware. CPU: 10.1 FPS (98.9ms). Intel iGPU: 15.2 FPS (65.7ms). That is a 1.5x speedup from OpenVINO hardware-aware inference. RESULTS 100% task success across all five tasks, evaluated over 10 random seeds each. Average episode: 52 steps, 3.3 seconds. Voice-to-motion latency under 1 second. SUPPORTED TASKS pick_place, drawer_open, drawer_close, door_open, button_press INTEL TECHNOLOGIES OpenVINO converts and runs the vision encoder on Intel CPU and iGPU. The pipeline targets Intel Core Ultra Series 2/3 hardware with NPU support in the benchmark runner. All inference is local — no cloud GPU required.

Table for Two: Bimanual SO-101 on OpenVINO

Table for Two turns one instruction, typed or spoken (Speechmatics), into a two-arm table-setting routine in MuJoCo: both arms carry the plate to the placemat, the left arm opens the drawer while the right places the mug, the fork is laid, water is poured into the mug, the bottle goes back, and the spoon is handed from one arm to the other. Nothing is attached to a gripper. Objects move only because the finger pads squeeze them and friction holds them; the gripper uses the stock servo's sustainable torque, the water is 24 beads, and mass and friction are randomised per scene. A Qwen2.5-1.5B INT4 model on OpenVINO GenAI writes the plan in a compact plan language, and every plan is checked (syntax plus a simulation of what each hand holds) before the robot moves: 20/20 paraphrased instructions give valid plans. A LeRobot ACT policy (4 cameras, joint positions and velocities, subtask token) trained on 280 scripted contact demos drives the arms; NNCF INT8 runs in 4.9 ms per call on an Intel CPU, 3.4x faster than FP32. Results on 10 randomised scenes: the scripted pipeline sets the whole table 10/10 (29/30 on unseen scenes). The learned policy does the plate, drawer and mug 10/10 and the fork 9/10, but the pour, bottle return and hand-over still break the chain, so it has not set a full table on its own; from a clean start it returns the bottle 10/10 and hands the spoon over 5/10. In the video those three stages are run by the scripted skills and labelled on screen. Every number, the checkpoint and the OpenVINO models are in the repository and its release.