What you'll learn
Quick Answer
Speech recognition (ASR) converts a raw audio waveform into a spectrogram-like representation, then feeds it through a neural network, usually a transformer or Conformer encoder plus a decoder, trained end-to-end to output text directly, no separate phoneme-matching pipeline anymore. Text-to-speech runs largely in reverse: a sequence model converts text into a mel spectrogram, then a separate neural vocoder converts that spectrogram into an actual audio waveform sample by sample. This post is conceptual and explanatory since there's no audio hardware in this environment to run live models, every mechanism described is verified against current, real documentation and papers.
Turning sound into something a network can use
A microphone produces a raw waveform, air pressure sampled thousands of times per second (commonly 16,000 samples per second for speech). Feeding that raw waveform directly into a neural network is technically possible but wasteful: a few seconds of audio is already tens of thousands of numbers, most of which carry redundant information a human ear wouldn't even distinguish.
Almost every speech system instead first converts the waveform into a spectrogram, a representation of how much energy exists at different frequencies, at each small time window, computed with a sliding Fourier transform. Speech systems specifically use a mel spectrogram, which warps the frequency axis to match how human hearing actually perceives pitch, we're far more sensitive to differences at low frequencies than at high ones, so the mel scale compresses high frequencies and expands low ones, matching perceptual sensitivity rather than treating every frequency band as equally important. This representation is both far more compact than the raw waveform and already organized around exactly the acoustic features, pitch, formants, energy over time, that carry the actual linguistic information.
Both speech recognition and text-to-speech systems are built around this same mel spectrogram representation, just moving in opposite directions: ASR converts audio into a mel spectrogram and then into text; TTS converts text into a mel spectrogram and then into audio.
How modern ASR maps sound to text
Older speech recognition systems (broadly, the era before roughly the mid-2010s) worked as a multi-stage pipeline: an acoustic model matched short audio segments to phonemes (the distinct units of sound in a language, roughly 40-44 of them in English), a pronunciation lexicon mapped sequences of phonemes to words, and a separate language model scored which word sequences were actually plausible. Each stage was trained somewhat independently, and getting good results meant tuning the interaction between all three.
Modern ASR systems are end-to-end: one neural network, trained directly on paired audio and text transcripts, learns everything, acoustic patterns, pronunciation, and word-sequence plausibility, jointly, without a hand-built pronunciation lexicon or separately trained language model stage. The obstacle standing in the way of "just train a network on audio-to-text" is that audio and text aren't aligned: a spoken word might take anywhere from a fraction of a second to over a second, and you don't have a labeled record of exactly which audio frame corresponds to which letter or phoneme. CTC (Connectionist Temporal Classification) is the specific technique that solved this: it's a loss function that lets a network output a probability distribution over characters (or sub-word units) at every single time step, including a special "blank" output for "nothing new was said yet," and sums the probability over every possible valid alignment during training, so the network learns to align audio to text on its own, without ever being told the exact time boundaries.
Today's strongest ASR models mostly use a Conformer architecture for the acoustic encoder, a hybrid that combines a transformer's self-attention (good at capturing long-range context across the whole utterance) with convolutional layers (good at capturing the fine, local acoustic detail convolution is naturally suited to, similar in spirit to how a CNN handles local image structure). The encoder's output then feeds either a CTC decoder, an attention-based decoder, or both combined, to produce the final text.
Whisper as a concrete, real example
OpenAI's Whisper is a useful concrete case because its architecture is fully public and simple to describe: it's a standard encoder-decoder transformer, the same general architecture family used for machine translation, adapted for audio. Input audio is chopped into 30-second chunks, converted to a log-mel spectrogram, and fed into the transformer encoder, which processes it through stacked self-attention and feed-forward layers to build up a representation of what was said acoustically. The decoder then generates the output text autoregressively, one token at a time, attending back to the encoder's audio representation and to the tokens it has already generated, exactly like how a text-generation transformer predicts the next word conditioned on everything before it.
What makes Whisper notable isn't a novel architecture, it's scale and multitask training: it was trained on 680,000 hours of multilingual, multitask audio scraped from the web with its existing (often imperfect) transcripts, rather than a smaller, carefully curated dataset. The same model handles transcription, translation into English, language identification, and voice activity detection, because all of these are framed as "predict the right sequence of tokens" and mixed into training, a single decoder replacing what used to be several separately engineered pipeline stages.
How text-to-speech builds audio in reverse
Neural TTS splits the text-to-audio problem into two separate stages, and this two-stage split (rather than text directly to waveform samples in one model) has stayed the dominant approach for a specific reason: predicting a mel spectrogram is a much shorter, lower-resolution sequence problem than predicting raw audio samples directly, and a separate, highly optimized second model can specialize purely in the harder job of turning that spectrogram into a natural-sounding waveform.
The first stage, exemplified by Tacotron, is a sequence-to-sequence model with an attention mechanism, structurally similar to the encoder-decoder ASR models above but running the opposite direction: an encoder maps the input text (characters or phonemes) into a representation, and an attention-based decoder generates a mel spectrogram, frame by frame, learning on its own how to align, through attention weights, which part of the input text corresponds to which moment of output audio, including things like how long to draw out particular sounds.
The second stage is the vocoder, a model whose only job is converting that predicted mel spectrogram into an actual, playable waveform. The original, defining example is WaveNet: a model that generates one raw audio sample at a time, each new sample predicted conditioned on the mel spectrogram plus every sample it has generated so far, using dilated convolutions to let each prediction see a long history of prior samples efficiently. "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (the actual Tacotron 2 paper title) describes exactly this pairing: Tacotron's predicted mel spectrogram feeding a WaveNet vocoder to produce the final audio. Generating one sample at a time at 16,000 or 24,000 samples per second is inherently slow, which is exactly why most production vocoders since have moved toward architectures that generate many samples in parallel instead of one at a time, trading some of the original WaveNet's fully autoregressive design for speed while keeping the same mel-spectrogram-to-waveform framing.
Why both directions are still genuinely hard problems
On the recognition side, the practical difficulties aren't really about understanding clean, single-speaker audio anymore, that's largely a solved problem for major languages. The remaining hard cases are background noise and overlapping speakers, strong accents and code-switching (common in Indian English, where speakers routinely mix languages mid-sentence), homophone ambiguity that only resolves with context ("recognize speech" versus "wreck a nice beach", a genuinely famous example in the field for a reason), and streaming constraints, a voice assistant can't wait for you to finish an entire sentence before starting to respond, so real-time systems have to produce partial, revisable hypotheses as audio arrives, a much harder constraint than transcribing a pre-recorded file all at once.
On the synthesis side, the hard part is prosody, the rhythm, stress, and intonation that makes speech sound natural instead of flat or monotone. The same sentence read as a question versus a statement, or with emphasis on a different word, changes meaning and requires the model to make choices a plain text input doesn't fully specify. Homographs are a related headache ("I will read the book" versus "I have read the book" are spelled identically but pronounced differently), and getting this right in a system generating from raw text, without any additional annotation about intended meaning, is still an active area of active model improvement rather than a fully closed problem.
Where the two meet in a real voice assistant
A practical voice assistant chains several distinct stages together, not just ASR and TTS alone. Voice activity detection (VAD) runs continuously and cheaply to decide when someone has actually started and stopped speaking, so the more expensive ASR model only runs on actual speech rather than constant background noise. ASR then converts that speech to text, an NLU (natural language understanding) or, increasingly, a general LLM interprets the text and produces a response, and TTS converts that response back into audio to play back.
Each stage historically added its own latency, run one after another, which is part of why early voice assistants felt sluggish for anything beyond a short command. The current direction of research and product development is toward tighter integration, some newer systems process speech input and generate speech output within a more unified model rather than as fully separate, sequentially chained ASR-then-LLM-then-TTS stages, aiming to cut the accumulated latency and also preserve information a text-only intermediate step throws away, like tone of voice or emotional emphasis. That is a genuinely active, fast-moving area rather than a settled architecture, which is exactly why this post focuses on the ASR and TTS mechanisms that are well-established and verifiable today rather than speculating on where unified speech-to-speech models will land.