Back to Study Material Artificial Intelligence

Speech recognition

Speech recognition, also called automatic speech recognition or ASR, is the task of transcribing spoken audio into text. It is a challenging AI problem bec...

Speech recognition

Converting Sound to Text

Speech recognition, also called automatic speech recognition or ASR, is the task of transcribing spoken audio into text. It is a challenging AI problem because spoken language is highly variable: the same word sounds different across speakers, accents, speaking rates, emotional states, and acoustic environments. Background noise, reverberation, and overlapping speakers add further complexity.

Early speech recognition systems were built using Hidden Markov Models, or HMMs, combined with Gaussian mixture models. The HMM modelled temporal sequences of acoustic states corresponding to phonemes, the smallest distinctive sound units of a language. A pronunciation dictionary mapped words to sequences of phonemes, and a language model assigned probabilities to word sequences to resolve acoustic ambiguity. This pipeline approach dominated the field for decades and achieved commercially deployable accuracy for constrained vocabularies.

Deep learning transformed speech recognition dramatically, beginning with the adoption of deep neural networks to replace Gaussian mixture models for acoustic modelling around 2012. End-to-end models like Connectionist Temporal Classification, CTC, and sequence-to-sequence models with attention removed the need for hand-crafted pronunciation dictionaries by learning the mapping from audio directly to text characters or subword units. OpenAI's Whisper model, trained on six hundred and eighty thousand hours of labelled audio, achieves near-human accuracy across ninety-nine languages without any language-specific adaptation.

Word Error Rate

The standard evaluation metric for speech recognition is Word Error Rate, or WER, defined as the number of substitutions, deletions, and insertions needed to convert the hypothesis transcript into the reference transcript, divided by the number of words in the reference. Human performance on clean speech is approximately five to eight percent WER depending on the domain. State-of-the-art models now approach this level on standard benchmarks.

Interview Tip

Be ready to explain the acoustic model, language model, and decoder components of a traditional ASR pipeline and contrast them with end-to-end approaches. A key advantage of end-to-end models is that they can be fine-tuned on new domains with limited data because no separate pronunciation dictionary or language model needs to be updated. Knowing this trade-off between interpretability and adaptability is relevant for applied AI engineering roles.