What you'll learn
Quick Answer
Audio compression gets most of its size reduction by throwing away sound the human ear genuinely can't perceive — a psychoacoustic model finds frequencies masked by louder nearby frequencies and allocates them fewer or zero bits — before any general-purpose entropy coding even runs. MP3, AAC, and Opus all use this idea but differ in filter-bank design, efficiency, and target use case: MP3 maxes out at 320 kbps and struggles above 16 kHz at low bitrates, AAC gets roughly 30% better efficiency and wider format support, and Opus was purpose-built for real-time use with algorithmic delay as low as 5 ms.
Why Audio Compression Is Mostly Lossy, and That's the Point
General-purpose file compression (covered in our How File Compression Actually Works: Huffman Coding and the ZIP Format post) squeezes data by finding and removing statistical redundancy — repeated patterns, uneven symbol frequencies — without throwing away any actual information; you get the exact original bytes back. Run that same approach on raw audio and you get maybe a 10-20% reduction, because audio waveforms don't have that much redundancy in the Huffman-coding sense. Lossy audio codecs like MP3, AAC, and Opus get to 10x or more reduction by doing something fundamentally different first: deciding, based on a model of human hearing, which parts of the signal you literally cannot perceive, and discarding those before any entropy coding happens at all. The entropy-coding step from that earlier post still runs at the very end of the pipeline, on whatever's left — but the big win in audio compression happens one step earlier, at the perceptual-modeling stage, which general file compression has no equivalent of.
Psychoacoustic Modeling: Removing Sounds You Can't Hear
The core technique is called masking, and it comes in two forms. Frequency masking is the effect where a loud tone makes quieter tones at nearby frequencies inaudible — if there's a loud 1 kHz tone playing, a much quieter 1.1 kHz tone right next to it effectively disappears for a human listener even though it's still physically present in the signal. Temporal masking is the same idea across time: a loud sound briefly masks quieter sounds just before and just after it, because the ear's response doesn't snap back instantly.
To exploit this, the encoder runs a Fast Fourier Transform on the signal to find where energy is concentrated, then computes a masking threshold for each "critical band" — frequency ranges that roughly match how finely the human ear itself can actually resolve pitch, which get narrower at low frequencies and wider at high frequencies, mirroring real auditory physiology rather than splitting the spectrum evenly. Wherever the masking threshold is high (a loud sound is covering that band), the encoder allocates fewer bits to representing that part of the signal precisely, because quantization noise introduced there will itself be masked and inaudible. Wherever masking is low (a quiet, exposed part of the spectrum), it spends more bits to keep the noise below what the ear could actually detect. The noise doesn't vanish — it's mathematically still there in the decoded signal — it's just engineered to stay beneath the threshold where you'd notice it.
MP3: Frames, Bitrates, and Its Real Limitations
An MP3 file is a stream of independent frames, each one self-contained enough to be decoded on its own: a header (starting with a sync word that lets a decoder or player find frame boundaries even if it starts reading mid-stream), side information describing how that frame's data is organized, and the compressed audio data itself.
Bitrate is a setting you choose before encoding, and it directly trades file size against quality: 128 kbps is roughly speech/preview quality, and 320 kbps is the highest bitrate the MP3 format supports at all — there's a hard ceiling, unlike some newer formats. MP3 also supports variable bitrate (VBR), where the actual bits-per-second shifts frame to frame based on how complex that moment of audio is, spending more bits on dense passages and fewer on simple ones, which gets better average quality per total file size than a fixed constant bitrate (CBR) at the same average rate. MP3's well-documented weakness shows up above roughly 16 kHz at lower bitrates: its hybrid filter bank (a 32-band subband filter combined with a further transform) introduces distortion that becomes audible as smearing or outright loss of high-frequency content exactly where it's already most data-expensive to represent — a structural limitation of the format's filter-bank design from the early 1990s, not something later encoders can fully engineer around.
AAC: What Actually Improved Over MP3
AAC (Advanced Audio Coding), standardized as MP3's intended successor, changes the filter bank design: it uses a pure Modified Discrete Cosine Transform (MDCT) rather than MP3's hybrid subband-plus-transform approach, which avoids the specific distortion source described above. The documented result is roughly 30% better compression efficiency at equivalent perceived quality — concretely, a 128 kbps AAC file is generally considered comparable in quality to a 192 kbps MP3 file, meaning you can ship meaningfully smaller files for the same listening experience.
Beyond raw efficiency, AAC extends what the format can represent at all: it supports sample rates from 8 kHz up to 96 kHz, versus MP3's 48 kHz ceiling, and multichannel audio up to 48 channels (standard 5.1 surround and beyond), where MP3 was designed primarily around stereo. AAC's psychoacoustic masking calculations are also documented as more accurate than MP3's, and it handles transient sounds (sudden percussive hits, for instance) with less smearing, both direct consequences of the cleaner single-transform filter bank rather than incremental tuning of the same underlying design.
Opus: Built for Real-Time From Day One
Opus, standardized in RFC 6716, was designed from the ground up for interactive, real-time audio — voice calls, videoconferencing, in-game voice chat, live streaming — rather than being adapted from a format built for static file distribution. It achieves this by combining two different coding approaches internally: a linear-prediction coder (descended from the speech-focused SILK codec) tuned for voice, and a transform coder using MDCT (descended from the music-focused CELT codec) tuned for general audio, with the encoder able to blend or switch between them depending on what kind of signal it's actually compressing.
The headline real-time numbers, straight from RFC 6716: Opus supports bitrates anywhere from 6 kbit/s up to 510 kbit/s in a single format (an enormous range compared to MP3's fixed options), and its default algorithmic delay is just 26.5 ms — which can be pushed as low as 5 ms by trading away some quality or bitrate, specifically for applications where round-trip latency matters more than anything else. It offers six selectable frame durations (2.5, 5, 10, 20, 40, and 60 milliseconds) so an application can pick its own point on the latency-versus-efficiency curve. This is exactly why Opus is the default audio codec in WebRTC: a video call where audio arrives 200ms late is unusable regardless of how good it sounds, so Opus was built to let an application choose low latency as the priority when it needs to.
Picking a Codec Today
In practice, the choice usually isn't close once you know what you're optimizing for. Opus is the right default for anything real-time or interactive — video calls, voice chat, live streaming — which is why it's baked into WebRTC as standard. AAC is the right default for on-demand file or streaming distribution where you want the best quality-per-byte and broad modern device support — it's what YouTube, Spotify, and Apple's ecosystem lean on. MP3 is still worth choosing specifically for maximum compatibility with old hardware and software that may not support anything newer, at the cost of the efficiency and quality-ceiling tradeoffs above. And when you need the original bits back exactly, none of these apply — that's what a lossless format like FLAC is for, trading file size for bit-perfect reconstruction, the same lossless-vs-lossy tradeoff our general file compression post covers in more depth.