OpenAI Whisper

The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Whisper has been trained on 680,000 hours of multilingual and multitask supervised data collected from the web. Whisper is Encoder-Decoder model. Input audio is split into 30-second chunks, converted into a log-Mel spectrogram, and then passed into an encoder. A decoder is trained to predict the corresponding text caption, intermixed with special tokens that direct the single model to perform tasks such as language identification, phrase-level timestamps, multilingual speech transcription, and to-English speech translation.

General
Relese date	September, 2020
Author	OpenAI
Repository	https://github.com/openai/whisper
Type	general-purpose speech recognition model

Start building with Whisper

We have collected the best Whisper libraries and resources to help you get started to build with Whisper today. To see what others are building with Whisper, check out the community built Whisper Use Cases and Applications.

Tutorials

👉 Discover more Whisper Tutorials on lablab.build

Boilerplates

Kickstart your development with a GPT-3 based boilerplate. Boilerplates is a great way to headstart when building your next project with GPT-3.

Python Whisper Boilerplate Whisper gpt3 email generator
Streamlit Whisper, GPT-3 Boilerplate Whisper gpt3 sentiment analysis
React app to upload audio file boilerplate Uploading audio file to an API endpoint
Whisper Flask API Boilerplate Whisper API with Flask
Whisper Flask API with GPT-3 Boilerplate Whisper API with Flask and GPT-3
Whisper Streamlit Boilerplate Automatic Speech Recognition using OpenAI's Whisper

Libraries

Whisper API libraries and connectors.

Whisper API Whisper API reference
OpenAI Node.js API library for Whisper The OpenAI Node.js library provides convenient access to the OpenAI API from Node.js applications
OpenAI Python Library for Whisper The OpenAI Python library provides convenient access to the OpenAI API from applications written in the Python language

ShiftSense: AI Shift Handover for Factories

In manufacturing, shift handovers are a hidden weak point. Outgoing technicians rush to pass on what happened: a vibrating spindle, a temporary fix, a part on order. Details get lost, and the incoming shift rediscovers problems the hard way, causing downtime, scrap and safety risks. ShiftSense is an AI handover copilot. At the end of a shift, the technician records a short voice note in their own language. ShiftSense transcribes it with Whisper running on AMD ROCm, pulls the shift's machine sensor logs, and scores each machine with an anomaly-detection model, so risk predictions come from data, not from a language model guessing. Gemma, served through vLLM on AMD Instinct GPUs in the AMD Developer Cloud, then combines everything into a structured briefing: machines at risk and why, actions completed, open tasks, and recommended next steps, each linked to the relevant page of the maintenance manual. The workflow runs on Evolus: a document extractor turns maintenance PDFs and work orders into structured data, a knowledge library lets the agent answer technician questions with sources, and a workflow distributes the briefing to the next shift. When the model is uncertain or a safety-critical issue appears, ShiftSense escalates to a supervisor instead of acting alone. Users: maintenance technicians, shift leads and plant managers. Impact: less time spent on handovers, fewer repeated diagnoses, and earlier detection of failing equipment. Because the models run on open weights on AMD hardware, plants keep sensitive production data under their control at a predictable cost per briefing.

KindredRelay: Keeping Families Connected

When a loved one is in hospital, one family member often becomes the link between the care team and everyone else. They are trying to listen, provide support and repeat updates to relatives across different cities and time zones. KindredRelay gives the family one shared place to understand what is happening. Caregivers can record conversations with permission, upload document photos or type updates, then review the text before sharing. Relatives can follow a timeline, ask questions and coordinate help. Important updates stay connected to their sources. Extracted facts must pass checks against quoted text. Answers display selected source sentences with the speaker and time. When discharge dates differ, both statements remain visible and the system highlights what needs clarification. Questions without supporting information can be saved for the care team. AMD Token Factory powers the language and vision model inference. DeepSeek-V4-Flash extracts candidate facts and selects sentences for answers. Gemma-4-31B-IT reads document photos. Speech transcription runs on the app server's CPU, while deterministic code checks quotations and numbers. A runtime view records model calls, latency and token usage. The prototype uses fictional care cases and supports family communication, with clinical decisions left to the care team. Our next step is a pilot to measure time spent relaying updates and repeated questions. The goal is simple: less for one person to carry, and more ways for everyone to help.

DubGuard: Evidence Backed Narration Review

In industrial training and maintenance videos, one wrong word can change an entire instruction. “Switch the power off” becomes “switch the power on.” A 15-minute wait becomes 50 minutes. Comparing transcripts can reveal these differences, but knowing whether a change is an acceptable paraphrase or alters a critical requirement takes closer review. DubGuard helps reviewers make that distinction, with evidence they can check. Users can explore predefined scenarios or add their own reference scripts and mark requirements that need particular attention, such as numbers, warnings, on/off instructions and the order of steps. They then record or upload English narration. DubGuard keeps the original audio and uses Whisper, running locally on the CPU, to produce a timestamped draft transcript. Rule-based checks flag possible deviations and areas of uncertainty for review. For an additional perspective, users can choose to enable semantic analysis through AMD Token Factory using DeepSeek-V4-Flash. This sends only the script, review policy and transcript text—not the original audio. DubGuard checks that the returned citations match exact passages in the source text, and treats the model’s findings as advisory. Reviewers can listen to the audio, inspect flagged passages, correct transcription mistakes, document their decisions and export the results. Organizational sign-off remains with an authorized human reviewer, and critical findings or unresolved evidence cannot be automatically approved. Built with React, TypeScript, FastAPI, SQLite and Whisper, DubGuard helps people review industrial narration more consistently and keep a clear record of their decisions. It supports human judgment; it does not certify that an instruction is safe.

CLARIO

300 million people live with a rare disease, most starting in childhood. Diagnosis often takes over five years: appointments, inconclusive tests, the same story retold. The clues may already exist: a forgotten symptom, a lab value buried in a PDF, a pattern no single visit revealed. What if we could connect those dots sooner? CLARIO connects patients with unexplained symptoms and the doctors looking for answers. Patients keep a diary in their own words, by voice or text, and upload lab reports. CLARIO links clues across months and asks the one question that would help most. Clinicians get a dashboard with the history, symptom and lab trends, rare conditions worth considering, and an evidence-backed brief. Every AI finding can be traced, corrected or rejected, and the brief is a draft until a doctor signs it. Privacy is engineered in: AES-256-GCM vault, Argon2id keys, pseudonymisation before any remote call, tamper-evident AI logs, X25519-sealed sharing, on-device transcription. Bigger isn't always better. Our LoRA-tuned Gemma 4 E4B reaches F1 0.70 on 300 disease-held-out cases (0.35 before tuning), with 100% valid JSON and 99.8% quote grounding. On 100 of those cases, scored the same way, it beats Gemma 4 31B: 0.69 vs 0.47. On AMD: Gemma 4 31B on AMD Token Factory wrote our 2,992-diary training set, reads lab PDFs with MinerU (48/48 rows) and drafts the brief. Our fine-tuned model runs offline on an AMD Ryzen CPU (F1 0.64), so the diary can stay on the patient's computer. Also tested on real patient text (PsyTAR). Every number links to a file in the repo. We don't replace doctors. We help the patient's whole story reach them, because five years is too long to wait to be heard.

SayAgain — catches wrong radio read-backs

Intelligent Industry. An operator radios "Close valve two-oh-three, line Bravo." The technician answers "Copy, closing two-oh-eight": a valve on another line, and sometimes nobody notices. SayAgain checks every read-back against the order and the plant registry and puts the one wrong word in front of the shift supervisor, who confirms or dismisses the card. A model on the AMD Token Factory (JSON mode) only points at the action, tag and line words. Plain Python decides: VERIFIED, MISMATCH, NO-READBACK or ABSTAIN. A failed call is NOT AUDITED, never a guess; every call leaves a receipt. The ladder rule was pre-registered in git (tag prereg-v1) for the audio row; using it on written rows is a logged deviation. On 89 held-out scripted exchanges, no rung passed a wrong read-back; an LLM judge passed 2. Gemma-4-31B-IT was the only model to meet the rule (2/48 false alarms, after one re-audit of failed gateway calls); a plain regex met it too on clean words. Exploratory synthetic speech (ElevenLabs voices, simulated radio, local Whisper) still passed 0, but false alarms rose to 16/48 (regex, no added noise); no human recording is measured. Who pays: the HSE/K3 manager. Indonesia's SMK3 (PP 50/2012) requires a safety management system, and the timestamped order, read-back, verdict and supervisor decision serve as audit evidence. Priced per radio channel; wedge: a free 30-day listen-only baseline at one mill. Built by one developer with Claude Code; every result re-derives offline (eval.py --rescore). Replay: https://sayagain.edycu.dev/replay/ · MIT.

PaperOps – The Glove-Proof Industrial Dispatcher

When a machine stops, every minute costs money. The technician needs three answers fast: what is wrong, which part, and where is it. Today that means searching heavy manuals and old ERP screens with dirty gloves in a loud hall. PaperOps removes the screen. The technician holds one button or a USB foot pedal and says what they see: "Press three is stuck, error E four oh two." Seconds later a thermal printer prints the repair ticket: safety step first, the steps to tick off, the right part and its exact shelf. How it works: speech-to-text runs on our server (faster-whisper). Gemma-4-31B-IT on AMD GPUs via the AMD Token Factory turns the report into a structured fault; if Gemma is slow or down, a fallback model on AMD (MiMo-V2.6-Flash) takes over and the ticket names the model that answered. Manual, stock and shelf are looked up in code, so the safety step always comes from the manual and the model cannot invent parts. Unclear reports get a question ("Which machine?") instead of a guess; prompt injection is ignored. PaperOps also spots repeat faults ("3rd time in 12 days - check pump P2"), shows supervisors a Machine health view, and every ticket has a QR code to "Mark as fixed" on a phone. Measured on 15 test cases with Gemma alone: 15/15 correct tickets (AI step median 2.1-3.4 s), safety step first on 100 %, shelf location on every ticket that needs a part. Voice to printed ticket on the live server: typically 5-10 s. Users: maintenance technicians, shift leads and planners. Gains: time, safety, first-time fix, fewer repeat breakdowns. Live demo, no login: https://paperops.zozinnovations.com Worker demo: https://paperops.zozinnovations.com/?mode=floor Video: https://www.youtube.com/watch?v=oS-NslNx5fo Code (MIT, Docker): https://github.com/zozinnovations/paperops - demo data is fictional.