Speech to text model
Speech to text models, also called automatic speech recognition systems, convert an audio waveform into a sequence of words. The core challenge is that spo...
Fundamentals of Automatic Speech Recognition
Speech to text models, also called automatic speech recognition systems, convert an audio waveform into a sequence of words. The core challenge is that spoken language varies in accent, speed, background noise, and pronunciation in ways that make the mapping from sound to text highly ambiguous. Early systems modeled this probabilistically by combining acoustic models, which describe how phonemes sound, with language models, which describe which word sequences are likely.
Modern end-to-end neural approaches learn to map audio features directly to text sequences without the separate acoustic and language model components. The audio is typically preprocessed into a spectrogram or mel-filterbank features, which represent the audio as a time-frequency matrix. The neural network, often a convolutional or transformer-based encoder, produces intermediate representations that are then decoded into text.
Exam tip: The connectionist temporal classification loss, abbreviated CTC, is a key technique in end-to-end speech recognition training. CTC handles the alignment mismatch between audio frames and output tokens by summing over all valid alignment paths between the input and output sequences, making it possible to train without frame-level labels. Be able to describe what problem CTC solves in one or two sentences.
Whisper and Transformer-Based ASR
OpenAI's Whisper model demonstrated that training a transformer encoder-decoder on large-scale weakly supervised audio-transcript pairs collected from the internet produces state-of-the-art speech recognition across many languages and conditions without task-specific fine-tuning. The encoder processes mel spectrogram features of 30-second audio segments, and the decoder autoregressively generates the transcript tokens, conditioning on the encoded audio through cross-attention.
Whisper's training data included audio in dozens of languages, which naturally produces a multilingual model without any explicit language segmentation during training. The model uses a special language token prepended to the decoder input to condition the output language, and a no-speech token probability to detect silent or non-speech audio segments.
Large-scale weakly supervised pretraining of this type, using internet-scale data with noisy labels rather than expensively curated clean transcripts, has proven effective across many speech tasks and has reduced the need for high-quality human-labeled speech datasets as a prerequisite for building high-quality ASR systems.
Streaming, Word Error Rate, and Real-World Challenges
Word error rate, abbreviated WER, is the standard evaluation metric for speech recognition. It counts the minimum number of word insertions, deletions, and substitutions needed to transform the model's output into the reference transcript, divided by the number of words in the reference. Lower WER indicates better accuracy, with near-zero WER indicating near-perfect transcription. Benchmark WER figures below five percent are routinely achieved by modern models on clean read-speech datasets, though performance degrades on spontaneous or accented speech.
Streaming speech recognition requires producing partial transcripts in real time as audio arrives rather than waiting for a complete audio segment. This constrains the model's ability to use future context, which typically degrades accuracy compared to offline processing. Many real-time systems use a two-pass approach: a first pass produces a streaming hypothesis quickly, and a second pass refines it with full context once the utterance is complete.
Channel conditions including microphone quality, room acoustics, and background noise affect real-world WER significantly. Systems deployed in noisy environments such as call centers or outdoor settings often require domain-specific fine-tuning on representative noise conditions to achieve acceptable accuracy.