Blogs / Systems

Choosing a Speech-to-Text Model: Why Whisper When Gboard Already Works Offline?

By Kadam Parikh · · 15 min read · Based on papers, model cards and leaderboards

I asked around 20 AI engineers why anyone would use Whisper when Google’s keyboard already dictates offline on a phone. None of them could give me a clear answer. This post gives the answer I found, and a way to choose a speech model, whether you are building the feature or asking for it.

The puzzle

Gboard, Google’s keyboard, types what you say as you say it, without internet, on a phone. Whisper is much bigger and does not obviously type as you talk. So why would anyone reach for it? When I put this to engineers, I got “Whisper is more accurate” or “it depends”, without numbers. Most teams pick the model they already know. Whisper is open, easy to run and covers many languages, and nobody gets blamed for the famous choice. That is understandable, but it is not a method. This post offers one.

One note on scope. All of this is about speech to text: audio in, written words out. Many of the 20 to 50 MB models in social-media demos do the opposite, text to speech, and comparing those with Whisper says nothing about Whisper.

Why this is not only an engineering question

Speech to text usually starts as a request, not a research project. A customer asks for it, or a founder or product team says “we want voice in here.” The usual answer is “that’s simple, a day or two.” It sounds fair, because a demo really can work in a day or two: pick a well-known model, send it some audio, get text back.

The demo isn’t what you asked for, though. Nobody in that conversation asked the questions that decide whether the feature works for your users. Do words need to appear while people speak, or after? Does it run on their phone or on your server? Which languages do they speak, and how should the text look? Does the audio have to stay on the device? The famous model gets built because it is the easy choice, and the answers to those questions are found out later: in a slow release, a surprise cloud bill, or a customer saying it doesn’t understand them.

So this post is for the people who ask for the feature as much as the people who build it. If you are a founder, product manager, CTO or client, the three questions below are the ones to ask before you accept an estimate. If you are an engineer, they are the ones to answer before you start.

Three questions, then a test

Most choices come down to three questions, asked in order. Each one removes options. Questions 1 and 2 come with the ideas that explain why, and question 3 comes down to checking your own language. They are also the questions to put to whoever gives you an estimate.

  1. Question 1

    • Live or recorded?words as someone speaks, or a finished file
  2. Question 2

    • Where does it run?phone, laptop or server
  3. Question 3

    • Which languages?and what should the text look like
  4. Then

    • A shortlisttwo or three candidates and a baseline
  5. Finally

    • A test on your audioerror, delay and memory, on your device
The path. Questions 1 to 3 narrow the choice. The test makes the final decision.

Question 1: is it live or recorded?

If words must appear while someone speaks, you need a model that streams. If you transcribe a finished recording, you do not. Two ideas explain why.

Idea: a model listens, then writes

Almost every speech model has two jobs, in order. It listens, turning a stretch of audio into an internal summary of the sounds. Then it writes, turning that summary into words one after another, each depending on the ones before. Models differ in how much audio they listen to before they start writing.

  1. 1 · Audio in

    • Whisper: a 30-second windowShort clips are padded with silence to fill it
    • Streaming: a small piecefor example 80 ms to about a second
  2. 2 · Listen (encoder)

    • Whisper: reads the whole windowonce, before any text exists
    • Streaming: reads the new pieceand keeps a short memory of the earlier ones
  3. 3 · Write (decoder)

    • Whisper: writes the wordsone after another, each using the ones before
    • Streaming: writes what it can so farthe newest words may be less accurate
  4. 4 · Text out

    • Whisper: appears after the clipafter it has been heard, not while it is spoken
    • Streaming: appears while you talkthen the next piece arrives and it repeats
Top row: Whisper. Bottom row: a streaming model. Simplified: this leaves out how long recordings are split into windows.

Whisper’s paper says it is trained on 30-second chunks and “cannot consume longer audio inputs at once”, so it listens to a window and then writes. A streaming model listens to a small piece, writes, and repeats. Google’s 2019 phone model is the streaming kind, and its blog says it “outputs characters one-by-one, as you speak”. Whisper can be wrapped to work live, but the authors of the best-known wrapper say it “is not designed for real time transcription”. Their wrapper reports an average delay of 3.3 seconds, measured on a server graphics card. I found no published delay for it on a phone.

Idea: two clocks, not one

Listening and writing take time, and that time can be measured two ways. Say you talk for a minute. One clock measures how long the machine works after you stop. The other measures how long each word takes to appear while you are still talking. The table uses invented numbers to show how far apart they can be.

You talk for 60 secondsWords appearWork after you stopFeels live?
Engine A: waits for you to finish, then worksAll at once, after 60 s6 sNo
Engine B: works as you talkAbout 1 s after each wordUnder 1 sYes

Illustrative numbers, not measurements.

Engine A is quick once you stop, but you stare at a blank screen for a minute. Engine B feels like dictation. Microsoft’s documentation for its speech engine draws the same line between the real-time factor (processing time divided by audio length) and user-perceived latency (the time between a word being spoken and shown).

For question 1: if the job is live typing, keep only models built to stream, and ask for the delay, not the real-time factor. If it is recorded audio, delay does not matter, and Whisper-style models stay in play.

Question 2: where does it run?

A phone, a laptop and a server are different machines. A model that is quick on a laptop can be slow on a phone, and a phone also slows down as it heats up: a 2026 preprint found an iPhone 16 Pro lost nearly half its throughput within two iterations of a sustained load, though it tested a language model, not a speech model. So the model must be timed on the device you will ship on. File size does not tell you what a device needs: the whisper.cpp README lists about 273 MB of memory for Whisper Tiny, whose quantized file is 32 MB.

Idea: size is not speed

Parameters count what a model stores, not how much work it does for each second of audio. A Whisper model does its work in two stages that repeat different amounts. The encoder listens, and runs once for each window. The decoder writes, and runs once for every word.

  1. 1 · Audio in

    • One window30 seconds; shorter clips are padded to fill it
  2. 2 · Encoder: listen

    • Runs oncefor the whole window
  3. 3 · Decoder: write

    • Runs once per wordevery word is another pass
  4. 4 · Text out

    • The wordsthe decoder stops at the end of speech
Where the work goes. Simplified: this leaves out the internal steps inside each stage.

So the writing stage carries most of the repeated work, and shrinking it saves time on every word. That is what Whisper Turbo does: it is Whisper Large with “the number of decoding layers reduced from 32 to 4”. It has about half the parameters of Large and, in OpenAI’s own English test on an A100 graphics card, runs about eight times faster (see the README). Moonshine saves work another way: it takes audio of any length without padding, so its smallest model has 27.1 million parameters against Whisper tiny.en’s 37.8 million and needs 0.7 times the compute, by its own paper’s table.

For question 2: do not rank models by size. Ask where a model does its work, and time it on your device, for as long as a real user would use it. A quantized Whisper Tiny (32 MB) or Base (about 60 MB) is small enough for a phone. Medium and Large are not.

Question 3: which languages, and what should the text look like?

This question most often changes the answer. On clean English, Whisper sizes are close: on read speech the paper gives 3.1% error for Small.en and 2.7% for the multilingual Large-v2, though 5.6% for Tiny.en, about double. Other languages can be far less forgiving, and the smallest models can fail outright. Hindi is the example here: Whisper’s multilingual models on the FLEURS test set, from the same paper, next to NVIDIA’s streaming model.

ModelHindi error on FLEURSStreams?
Whisper Tiny102.3%No
Whisper Base101.1%No
Whisper Small38.4%No
Whisper Large-v221.5%No
Nemotron 3.5, language given6.8% to 8.1%Yes

FLEURS is read speech, not notes. The two sources use different harnesses, so the gap is large but not a controlled comparison. The Whisper figures are for Large-v2; the paper reports no later version. An error above 100% is possible because extra words count as mistakes: 14 wrong words against a 10-word reference scores 140%.

Your language will have its own numbers, so look them up, or measure them in the test.

The text can also come out in different forms. Take a Hindi speaker saying “today is the design review”, with the English words “design review” in the middle of the sentence. A model can write it three ways:

Each model supports different forms, so decide which you need before you test. Whisper’s translation mode outputs English only, so it cannot produce the mixed form (README).

For question 3: write down your languages and the text form you need, and find each candidate’s error for your language. If it is not clean English, keep every size of a candidate in play until the test, instead of assuming the smallest will do.

Make a shortlist

Take two or three candidates that fit your answers, and one baseline: the model you would ship if the candidates fail, and the yardstick that tells you whether they were worth the trouble. These rows are my judgment, not a published guideline.

SituationCandidates to testBaseline
Live, English, on a phoneMoonshine streaming Tiny or Small; sherpa-onnx streaming Zipformer; Vosk smallWhisper Tiny or Base, quantized (expect more delay)
Live, a language other than English, on a phoneNemotron 3.5 streaming (many languages, about 740 MB); Vosk small models, where one exists for your language; for Hindi, Salesken (460 MB, no published accuracy)Whisper Small, quantized (not built to stream)
Recorded English, on a serverParakeet TDT v2 (English only; lower average error and much higher speed than Whisper large-v3 on the Open ASR Leaderboard)Whisper large-v3 or Turbo
Recorded, many languagesWhisper Medium or Large; Parakeet v3 if your languages are among its 25 European onesWhisper Small
Recorded, two languages mixed in one sentence (for example Hindi and English)For Hindi and English: Trelis tara (1.55B, 3.09 GB) and Srota-Conv (782M, 1.57 GB). Both report their own results, and neither streamsWhisper Small
Speech in one language, text in English (translation)Whisper Medium or Large, in translate mode (English output only)Whisper Small

How to run them. Moonshine ships its own library for iOS, Android, Linux, Windows and Raspberry Pi. sherpa-onnx runs the streaming Zipformer models, and Vosk has its own streaming library. Whisper runs locally with whisper.cpp. Parakeet, Nemotron and Salesken are NVIDIA NeMo models, and Nemotron also publishes an 8-bit file. I have not verified any of these on a phone.

Licence and privacy. Moonshine is MIT, Vosk and Salesken are Apache-2.0, tara is Apache-2.0, and Nemotron 3.5 is under OpenMDW-1.1, a permissive licence that asks you to keep its notices and a copy of the licence if you distribute the model (this is not legal advice). For the others, read the model card before you build on it. Every model here runs on your own device, so audio does not have to leave it. A hosted API would send it away.

Write the bar before you test

The bar is a number for each thing you care about: error, first-word delay, memory and licence. An invented example: Hindi error under 25% on my audio, first word within 1 second, under 500 MB of memory, and a licence that allows commercial use. The bar comes first. Among candidates that pass it, take the smaller one.

This is the step where the business decides. Product, the founder or the client knows what good enough means for their users, so they set the numbers. Engineering then measures against them. If nobody writes the bar, the team ends up judging the feature by whichever demo looked best.

Test on your own audio

Leaderboards answer someone else’s question, and they change: Moonshine was removed from the Open ASR Leaderboard in July 2026, and the same Whisper model (large-v3) was listed there at 145 on its speed scale in May and 469 in November, after the test machines changed. Your own test answers yours, and needs no code to design.

  1. Record about ten minutes in your real conditions: your speakers, microphone, noise and languages. Include a silent stretch.
  2. Write the correct transcript by hand for three minutes. Before counting errors, put both transcripts in the same form: same case, no punctuation, numbers written the same way.
  3. Run every candidate and the baseline on the device you will ship on. Measure error against your transcript. Measure delay by recording the screen while you say ten marker words, then take the median gap between each word and its appearance. Measure peak memory from the phone’s developer tools, not the file size.
  4. Run it for ten minutes in a row, because phones slow as they heat.
  5. Read the outputs. Look for invented sentences during the silence and for a phrase repeated over and over.

Compare each result with your bar. The smallest candidate that passes wins. If none passes, move to a bigger model or a different kind, or relax the bar.

Worked example: dictation in a notes app

Suppose you are adding dictation to a notes app on a mid-range Android phone, offline. The requirements are invented, and Hindi is only the example language; any language with its own script raises the same questions. The bar: Hindi error under 25%, first word within 1 second, under 500 MB of memory, a licence that allows commercial use, and English words kept in Latin script, as spoken.

  1. Question 1: live. Words must appear as the user speaks, so only streaming models qualify, and Whisper is a baseline, not a candidate.
  2. Question 2: a mid-range phone. A model’s file size is a floor on its memory use, so a file already over 500 MB fails before any test.
  3. Question 3: Hindi with English words in it, written in mixed script. Whisper Tiny and Base are out even as baselines, with Hindi error above 100%.
CandidateFilePublished Hindi resultEnglish words in the output
Nemotron 3.5 streamingAbout 740 MB at 8-bit6.8% to 8.1% with language given; NVIDIA’s cardNot stated on the card
Salesken Hindi streaming460 MBNo accuracy on its cardProbably Devanagari: 1,016 of its 1,023 vocabulary entries are Devanagari
Vosk small Hindi42 MB20.9% and 24.7% on two test sets, metric not statedNot stated on the Vosk page
Whisper Small, quantized (baseline)190 MB38.4% on FLEURS; not built to streamNot checked

The bar settles one thing without a test. Nemotron’s file is over 500 MB, so it fails. It does not settle the rest. Salesken’s file leaves almost no room once the model is running, and it publishes no accuracy. Vosk fits easily, but its published error sits near the edge of the bar. So test Vosk and Salesken against the Whisper baseline on your audio, and decide there. If neither passes, the choice is the one the path leads to: raise the memory limit and test Nemotron, or relax the script requirement. Nobody has published live Hindi dictation on a phone for any of these, so the measuring is yours.

Why testing is mandatory

The numbers you find online, in papers, model cards and leaderboards alike, are rarely the ones you actually need. They come from someone else’s audio, devices and settings, so there will be gaps that you have to fill yourself, with a test. These are the ones I found while writing this post.

Frequently asked questions

What should I ask before adding speech to text to my product?

Ask whoever gives you the estimate the three questions in this post: live or recorded, where it runs, and which languages and text form. Then ask what result counts as good enough, and who tests it on your own recordings. A demo with a well-known model answers none of these, so “it’s simple, a day or two” is a reason to ask them, not a reason to relax.

Can Whisper do real-time transcription?

Not natively. Whisper reads audio in 30-second windows. Wrappers such as Whisper-Streaming feed it short stretches of audio and reported an average delay of 3.3 seconds on a server graphics card, according to their authors. I found no published streaming delay for Whisper on a phone. For live typing, a model built to stream is a better fit.

Can Whisper run on a phone?

The smaller sizes can. Quantized Whisper Tiny is about 32 MB and Base about 60 MB on disk, and Medium and Large are too large to be practical. The file understates the memory needed while running: whisper.cpp lists about 273 MB for Tiny. I found no published phone benchmark of Whisper streaming, so time it on your own device.

Why not use Gboard’s speech model?

Because you cannot take it. Google’s 2019 blog describes a streaming on-device recognizer of about 80 MB after compression (the paper reports 120 MB for its word-piece model), launched on Pixel phones in American English. Gboard’s help page says advanced voice typing keeps the text you speak on the device, apart from two optional editing features, and needs a Pixel 6 or newer. But Google has not published the current model’s size or accuracy, and there are no downloadable weights for developers.

Terms used in this post

Speech to text (STT, ASR)
Turning spoken audio into written text. “ASR” (automatic speech recognition) means the same thing. Whisper, Moonshine and Gboard voice typing are all speech to text.
Text to speech (TTS)
The opposite direction: written text in, spoken audio out. The 20 to 50 MB models in many social-media demos are usually this, which is a different job from Whisper’s.
Error (word error rate)
The number of mistakes in a transcript (wrong, missing and extra words) divided by the number of words in the correct transcript. Lower is better. It can go above 100%, because extra words add mistakes without limit.
Real-time factor (RTF)
Processing time divided by audio length. An RTF of 0.5 means one minute of audio takes thirty seconds to process. It says whether an engine keeps up, not when the first word appears.
Streaming
Producing text while the person is still talking, in small pieces, instead of waiting for a finished recording.
Delay (latency)
How long after a word is spoken it appears on screen. This decides whether dictation feels live.
Parameters
The learned numbers inside a model. “39M” means 39 million. More parameters usually means a bigger file, but not always more work per second of audio.
Quantization
Storing a model’s numbers with fewer bits (for example 5 or 8 instead of 16 or 32) so the file is smaller and often faster, usually at a small cost in accuracy.
Encoder and decoder
The two halves of models like Whisper. The encoder turns a stretch of audio into an internal summary. The decoder then writes the words from that summary, one at a time.
Devanagari and Latin script
Devanagari is the script Hindi is usually written in. Latin script is the one English uses.
Hinglish
Hindi and English mixed within a sentence, as many Indian speakers talk. Written down, it can use Devanagari for the Hindi words and Latin script for the English ones.

Sources