Summary
Speech-to-text (STT) converts speech into text by sampling continuous sound waves at 16,000 numbers per second (16 kHz), transforming them into a 2D mel spectrogram aligned with human hearing, and using a Transformer encoder-decoder to predict text tokens sequentially. While cloud STT offloads processing to remote servers, on-device STT runs optimized, smaller models locally on the device—eliminating network latency, enabling offline operation, and keeping voice data private, at the cost of tighter compute and model size constraints.
Voice assistants, real-time captioning, and meeting transcription apps all rely on speech recognition. This post explains how acoustic sound waves are transformed into numbers, mel spectrograms, and finally text, examines the architectural trade-offs between cloud STT and on-device STT, and details why recognition errors occur. Model architecture descriptions are based on OpenAI Whisper’s public sources (the GitHub README and audio.py), accompanied by an actual run of mel spectrogram feature extraction.
What is speech recognition (STT)? How does sound become numbers?
Short answer: Microphones capture continuous sound vibrations, which are digitized into discrete numerical samples at fixed intervals, sliced into short overlapping windows, and converted into a 2D frequency representation known as a mel spectrogram.
- Sampling: Continuous changes in air pressure are measured at fixed intervals to produce a discrete sequence of numbers. Whisper’s
audio.pydefinesSAMPLE_RATE = 16000, capturing 16,000 amplitude values per second. - Short window slicing: Because speech characteristics remain relatively stable over brief durations, audio is segmented using 25 ms windows (
N_FFT = 400) advanced in 10 ms steps (HOP_LENGTH = 160). - Frequency analysis and mel filterbank: Each windowed segment is multiplied by a window function and processed with a Fast Fourier Transform (FFT) to extract spectral energy across frequencies. A mel-scale filterbank—designed to emulate human auditory perception, which is more sensitive to pitch changes at lower frequencies—is applied, followed by a logarithm. Whisper utilizes either 80 or 128 mel filters depending on the model. The resulting 2D log-mel spectrogram maps time on the horizontal axis and frequency bins on the vertical axis, with pixel intensity representing spectral energy.
- Whisper’s 30-second chunking: In
audio.py, audio is processed in standardized 30-second chunks containing 480,000 samples (N_SAMPLES), which yield exactly 3,000 mel frames (N_FRAMES, or 100 frames per second).

actual output (Python 3 · NumPy 2.5.3): the code below is a simplified NumPy-only demo using Whisper-style parameters (16 kHz, window 400, hop 160, 80 mels) on a 1-second 440 Hz test tone. It is not Whisper’s exact production code.
# Whisper-style log-mel front end, numpy only (sr=16000, n_fft=400, hop=160, n_mels=80)
import numpy as np
SR, N_FFT, HOP, N_MELS = 16000, 400, 160, 80
def hz_to_mel(f): return 2595 * np.log10(1 + f / 700)
def mel_to_hz(m): return 700 * (10 ** (m / 2595) - 1)
def mel_filterbank():
mels = np.linspace(hz_to_mel(0), hz_to_mel(SR / 2), N_MELS + 2)
bins = np.floor((N_FFT + 1) * mel_to_hz(mels) / SR).astype(int)
fb = np.zeros((N_MELS, N_FFT // 2 + 1))
for i in range(1, N_MELS + 1):
l, c, r = bins[i - 1], bins[i], bins[i + 1]
fb[i - 1, l:c] = (np.arange(l, c) - l) / max(c - l, 1)
fb[i - 1, c:r] = (r - np.arange(c, r)) / max(r - c, 1)
return fb
t = np.arange(SR) / SR # 1 second
wave = 0.5 * np.sin(2 * np.pi * 440 * t) # 440 Hz test tone
frames = np.lib.stride_tricks.sliding_window_view(wave, N_FFT)[::HOP]
spec = np.abs(np.fft.rfft(frames * np.hanning(N_FFT), axis=1)) ** 2
mel = np.log10(np.maximum(spec @ mel_filterbank().T, 1e-10)).T
print("samples:", wave.shape[0])
print("frames x fft bins:", spec.shape)
print("log-mel shape (mels x frames):", mel.shape)
print("loudest mel band:", int(mel.mean(axis=1).argmax()))
The actual output:
samples: 16000
frames x fft bins: (98, 201)
log-mel shape (mels x frames): (80, 98)
loudest mel band: 15
One second of 16 kHz audio equates to 16,000 discrete numbers. Without padding, sliding a 25 ms window every 10 ms yields 98 frames (Whisper applies padding so each 30-second chunk produces 3,000 frames, or 100 frames per second). Each frame maps to 80 mel values, forming an 80×98 2D “sound image.” The pure 440 Hz tone produced the highest energy concentration in a lower frequency band, mel band 15.
How does an STT model turn a spectrogram into text?
Short answer: A Transformer encoder converts the log-mel spectrogram into acoustic feature vectors, and the autoregressive decoder iteratively predicts the next text token based on these acoustic representations and previously generated tokens.
- Unified sequence-to-sequence model: According to the Whisper README, “A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline” (openai/whisper).
- Encoder and decoder interaction: The encoder processes the 2D log-mel spectrogram to generate high-dimensional acoustic feature representations. The decoder takes these acoustic features alongside previously decoded tokens to predict subsequent tokens one by one. For more details on tokenization concepts, see What are tokens and the context window? (Korean).
- Comparison with legacy pipelines: According to the HN discussion, older speech recognition systems typically relied on constrained grammars and acoustic templates (HN thread). In contrast, modern end-to-end architectures learn the complete mapping from raw audio representations to text within a single unified neural network.
What is the difference between cloud STT and on-device STT?
Short answer: Cloud STT runs large, high-capacity models on remote servers, whereas on-device STT deploys compact, optimized models directly on the client device. This local execution delivers lower latency, enhanced privacy, and offline capability, balanced against stricter hardware constraints on model size and processing power. The comparison below is qualitative and does not represent measured benchmarks.
- Latency: Cloud STT requires network round trips to transmit raw audio and return transcription results. On-device STT processes audio locally, eliminating network transit time entirely.
- Privacy: Cloud STT transmits user voice recordings to external servers. On-device STT processes and retains all audio data strictly within the local device.
- Offline availability: On-device STT operates seamlessly without internet access, whereas cloud STT depends entirely on an active, stable connection.
- Accuracy vs. model size trade-off: Cloud infrastructure supports massive parameter counts and extensive memory allocations. In contrast, edge and mobile devices must operate within strict memory, battery, and thermal limits, necessitating aggressive model compression.

There are three primary strategies to shrink models for on-device deployment:
- Architectural scaling: Decreasing the number of layers and hidden dimensions to reduce overall parameter count.
- Quantization: Storing model weights in lower-precision representations (e.g., INT8) to minimize memory footprints and compute overhead. For a detailed breakdown, see What is quantization? (Korean).
- Language scope pruning: Restricting linguistic coverage, as demonstrated by the English-only
.enmodels (tiny.en, base.en, small.en, medium.en) in the Whisper repository.
Published figures illustrating model scale. The title of the Hacker News post “Whistle: Speech to Text in 16.9 MB” (961 points) highlights a 16.9 MB model footprint (HN thread). The official Whisper README lists the following model parameters and memory footprints:
- tiny: 39M parameters, ~1 GB VRAM
- base: 74M parameters, ~1 GB VRAM
- small: 244M parameters, ~2 GB VRAM
- medium: 769M parameters, ~5 GB VRAM
- large: 1550M parameters, ~10 GB VRAM
- turbo: 809M parameters, ~6 GB VRAM
The relative speed metrics reported in the Whisper README were evaluated on an NVIDIA A100 GPU transcribing English audio, accompanied by the explicit caveat that “real-world speed may vary significantly.” These figures should not be construed as indicative of transcription speeds on mobile or consumer edge devices.
Why does STT make mistakes, and what should you watch out for?
Short answer: Speech recognition models degrade when live audio deviates from their training distribution—such as with unfamiliar accents, acoustic noise, or rare terminology—and can occasionally hallucinate plausible phrases during silent intervals.
- Accents and dialects: Phonetic variations underrepresented in the training corpus can lead to higher error rates.
- Acoustic noise and reverberation: Background sounds and room echo distort spectral patterns in the mel spectrogram, degrading feature extraction accuracy.
- Domain jargon and proper nouns: Low-frequency vocabulary items absent from general training corpora are frequently misrecognized as phonetically similar common words.
- Cross-lingual variation: As stated in the Whisper README, “Whisper’s performance varies widely depending on the language.” Word Error Rate (WER) serves as the standard evaluation metric measuring the proportion of incorrect words.
Q1. Why does the model output phrases like “Thank you” during silent intervals?
According to the HN discussion, some users mention hallucinations where phrases such as “Thank you” are generated repeatedly during periods of silence or ambient noise. The underlying cause of this specific artifact was not verified in this post.
Q2. Is Korean speech recognition as accurate as English?
Accuracy varies substantially across target languages. The Whisper README notes that recognition performance (WER) differs widely depending on the language evaluated. Specific quantitative benchmarks comparing Korean and English are beyond the scope of this post.
Q3. What are the key limitations of on-device STT?
The primary constraints stem from restricted model capacity and available on-device compute. Compressing a model to operate on edge hardware may reduce recognition fidelity relative to large cloud models (such as Whisper large at 1550M parameters), particularly in noisy acoustic conditions or on uncommon vocabulary, while battery consumption and thermal throttling remain ongoing considerations. The precise trade-off varies depending on the specific model architecture and deployment environment, and cannot be generalized with a single benchmark.