Whisper

Whisper is OpenAI’s open-source automatic speech recognition (ASR) model, trained on more than 5 million hours of audio, that transcribes speech in 99 languages and translates any of them into English. Released under the MIT licence in September 2022, it was one of the first frontier-quality AI models a developer could download and run on their own hardware instead of calling a paid API, and it remains the baseline every newer speech recognition tool gets compared against. For developers building voice agents, transcription tools, or subtitle pipelines, Whisper is usually the first model worth testing, since it works well out of the box across accents and background noise without any fine-tuning.

How Whisper works

Whisper uses an encoder-decoder transformer, a neural network architecture where one half listens to the audio and the other half generates the matching text.

Audio encoding: The model splits incoming audio into 30-second chunks and converts each one into a spectrogram, a visual representation of sound frequencies over time, which the encoder processes into a sequence of numerical features.

Multitask decoding: A single decoder handles transcription, translation, and language identification at once, predicting special tokens that tell it which task to perform and in which language, rather than needing a separate model for each.

Weak supervision at scale: Instead of relying only on carefully labeled datasets, Whisper was trained on hundreds of thousands of hours of audio scraped from the internet with imperfect transcripts, which is why it generalises so well to real-world accents and noisy recordings.

Model sizes: Whisper ships in sizes from tiny, at 39 million parameters, to large-v3, at 1.55 billion parameters, letting you trade accuracy for speed depending on whether you need real-time results or maximum precision.

Deployment options: You can run Whisper locally with the open-source Python package, use optimised community ports for faster inference, or call OpenAI’s hosted API and pay per minute of audio instead of managing infrastructure yourself.

What you can build with Whisper

Meeting and podcast transcription: Feed recorded audio through Whisper to generate searchable transcripts and show notes automatically, instead of paying a human transcription service.

Multilingual subtitle generator: Transcribe a video in its original language, then use Whisper’s translation mode to produce an English subtitle track without a separate translation step.

Voice agent front end: Pair Whisper with a text-to-speech model and an LLM to build a voice assistant that listens, understands, and responds, all running on infrastructure you control.

Call centre quality review: Transcribe customer support calls in bulk and search the text for compliance issues, common complaints, or coaching opportunities across thousands of recordings.

Accessibility captioning: Generate live or near-live captions for streamed video content, making it usable for deaf and hard-of-hearing viewers without relying on a paid captioning vendor.