Model Quantization Explained: Shrinking a Model Without (Mostly) Breaking It

Quick Answer

Quantization takes the numbers inside a trained model and stores them with fewer bits, usually converting 32-bit decimals to 8-bit whole numbers. This reliably makes the model file about 4 times smaller, and it often but not always makes it run faster too, because the speed part depends on whether the chip running it actually has fast hardware support for 8-bit math.

What Quantization Actually Is

A trained machine learning model is, underneath all the jargon, a very large pile of numbers called weights. A small dense layer in a neural network might be a 2048x2048 grid of them, over 4 million individual numbers for just that one layer. Normally those numbers are stored as float32, meaning each one takes up 32 bits, or 4 bytes. That precision lets a weight be something like 0.0398721, accurate to many decimal places.

Quantization is the process of converting those same numbers into a format that uses fewer bits. The most common target is int8: a plain whole number from -128 to 127, taking up 8 bits, or 1 byte. Going from 4 bytes per number to 1 byte per number is, by definition, a 4x reduction in storage. There is no approximation or luck involved in that part of the claim; it falls directly out of the bit widths.

What makes this possible at all is that you cannot just truncate a float to an int and call it quantized, because a weight like 0.0398 would collapse to 0. Instead, quantization picks a scale factor based on the actual range of values in that weight matrix, maps the real numbers onto the int8 range using that scale, and keeps the scale factor around so the numbers can be converted back (dequantized) when needed for the actual math.

On a 2048x2048 float32 weight matrix, a real measurement looks like this:

That is a single layer. A real model has dozens to hundreds of layers, and large language models have billions of parameters, so this 4x reduction is the difference between a model that fits on a phone or a single GPU and one that does not fit at all. This is why quantization shows up constantly in both on-device AI and in serving large models cheaply in the cloud.

A Real Demo: Quantizing a Weight Matrix With Numpy

Explaining quantization in the abstract is easy to nod along to without really believing the numbers. So here is the actual code, run for real, with the actual printed output below it, not a guess at what it should say.

import numpy as np

W = np.random.randn(2048, 2048).astype(np.float32) * 0.05
x = np.random.randn(2048, 512).astype(np.float32)

# FP32 baseline
fp32_bytes = W.nbytes
y_fp32 = W @ x

# INT8 quantization: per-tensor, symmetric
w_absmax = np.abs(W).max()
scale = w_absmax / 127.0
W_q = np.round(W / scale).astype(np.int8)
int8_bytes = W_q.nbytes

# Dequantize to check real accuracy loss
W_dequant = W_q.astype(np.float32) * scale
abs_err = np.abs(W - W_dequant)
y_quant = W_dequant @ x
output_rel_diff = np.abs(y_fp32 - y_quant).mean() / np.abs(y_fp32).mean()

The scale factor is the entire trick: take the single largest absolute weight value in the matrix, divide it by 127 (the top of int8's range), and that becomes the conversion rate between the float world and the integer world. Every weight gets divided by that scale and rounded to the nearest whole number to become its int8 version. Running this for real produced:

FP32 weight matrix: 16,777,216 bytes (16.00 MB)
INT8 weight matrix: 4,194,304 bytes (4.00 MB)
Size reduction: 4.0x smaller

Mean absolute weight value: 0.039906
Mean absolute quantization error per weight: 0.000495
Mean relative weight error: 1.241%
Mean relative error in final layer output: 1.146%

So for this layer, every weight is off by about 1.24% on average after being squeezed into int8, and once you run an actual matrix multiply with those rounded weights, the final output numbers are off by about 1.15% compared to doing the same multiply in full float32. That is the real, measured cost of quantization on this example: not zero, but small enough that most models tolerate it without a noticeable drop in what they can do.

The Accuracy You Give Up, and Why It Usually Does Not Matter

A 1.15% error sounds small until you remember a real model chains together dozens of layers, and errors from each layer could in theory compound. In practice they mostly do not compound into disaster, for a few concrete reasons worth understanding rather than taking on faith.

First, rounding errors in quantization are roughly symmetric around zero: some weights round up slightly, some round down slightly. When you multiply a long row of weights against a row of activations and sum the results (exactly what a matrix multiply does), these small positive and negative errors partially cancel each other out rather than all stacking in the same direction. That is a big part of why the measured output error (1.15%) stayed close to the measured weight error (1.24%) instead of exploding.

Second, neural networks are trained with regularization and are, by design, somewhat redundant and tolerant of small perturbations; a model that broke the moment any single weight moved by 1% would also break from ordinary floating-point rounding differences between CPUs and GPUs, which already happens today. Production-grade quantization also uses tricks this demo skipped for clarity: per-channel scales instead of one scale for the whole matrix (so a few unusually large weights do not blow up the resolution for everyone else), and calibration on a small sample of real input data to choose scales that minimize error on data the model will actually see.

Third, there is a stronger technique than what this demo used, called quantization-aware training (QAT). Instead of training a model in full precision and then quantizing it afterward (called post-training quantization, or PTQ, which is what the code above does), QAT simulates the rounding during training itself, so the model's weights get nudged toward values that survive quantization well. QAT typically recovers most of the accuracy gap that plain PTQ leaves on the table, at the cost of needing to retrain rather than just convert an already-finished model.

The honest summary: quantization is not free, it has a measurable, non-zero accuracy cost, and that cost is usually small enough to accept for the size and (sometimes) speed benefits, but “usually small enough” is a judgment call specific to what the model is used for, not a universal guarantee.

Does It Actually Speed Things Up? (The Part Most Explainers Skip)

Most quick explanations of quantization imply that because the numbers are 4x smaller, the model also runs roughly 4x faster. That is the theoretical ceiling, and it is frequently not what happens in practice, which is worth saying plainly because it is the part that gets glossed over.

A real benchmarking study on Android devices found that across different mobile deep learning libraries, measured INT8 speedup ranged from about 0.8x to 3.0x compared to FP32, well short of the theoretical 4x, and in some configurations INT8 inference was actually slower than FP32. A separate Redis engineering writeup on this exact question, titled plainly “does quantization speed up inference,” found similarly mixed results on standard CPUs: on a 4-core Intel Xeon, measured speedup ranged from only about 1.02x to 1.56x.

The reason is mechanical, not mysterious: speed only improves if the hardware has a genuinely fast execution path for 8-bit integer math. If the chip's circuits are built around doing math in 32-bit float lanes, an int8 value often has to be converted back up before the actual multiply-and-add happens, which adds overhead on every single operation instead of removing it. On hardware that does have native low-precision support, such as GPUs with dedicated int8 tensor cores, that WebSearch found int8 tensor cores can roughly double the throughput of fp16 tensor cores on the same chip, which is a real, substantial win.

The practical takeaway for anyone evaluating quantization for a real project: the smaller file size and lower memory footprint are close to guaranteed. The faster-inference claim is conditional on your specific hardware and runtime actually having a fast int8 path, and it is worth benchmarking on your real target device rather than assuming the textbook 4x number, because the gap between “smaller” and “faster” is exactly where a lot of real-world quantization projects go wrong.

Post-Training Quantization, QAT, and Dynamic vs Static

There are a few named variants worth knowing by name, because they show up in any serious framework (PyTorch, TensorFlow Lite, ONNX Runtime) and interviewers sometimes ask which one you would pick and why.

Post-training quantization (PTQ) is what the demo above did: train a model normally in full precision, then convert the finished weights to int8 afterward. It is fast, needs no retraining, and is the first thing almost every team tries. Its downside is exactly the accuracy cost discussed above, uncorrected for.

Quantization-aware training (QAT) inserts fake quantization operations into the training loop itself, so the model's gradients are computed as if the weights were already going to be rounded, nudging the model toward weight values that are robust to that rounding. It needs the original training pipeline and more compute time, but it closes most of the accuracy gap PTQ leaves behind, and is the standard choice when a team is quantizing aggressively (down to 4-bit) or deploying into an accuracy-sensitive product.

Dynamic quantization only quantizes the weights ahead of time; the activations (the intermediate outputs flowing through the network at inference time) are quantized on the fly, computed fresh for each input. This is simple to apply and needs no calibration data, but carries a small runtime overhead for computing those activation scales every time.

Static quantization quantizes both weights and activations ahead of time, using a calibration dataset (a small batch of representative real inputs) to decide fixed activation scales in advance. It is faster at inference time than dynamic quantization because nothing needs to be computed on the fly, but it depends on the calibration data actually resembling real production traffic, or the fixed scales will be wrong for inputs that look different.

For a student project or a first attempt at shrinking a model, PTQ with dynamic quantization is the standard starting point: it is a few lines in most frameworks and needs nothing beyond the already-trained model.

Where This Shows Up in Real Products

Quantization is not a niche academic trick; it is one of the main reasons modern AI products are affordable to run at all. Two concrete, well-documented examples are worth knowing.

The first is on-device AI (covered in more depth in a companion post on on-device machine learning): a speech-to-text or keyboard-prediction model has to fit inside an app that is also doing a hundred other things on a phone with limited RAM. Shrinking that model 4x via int8 quantization is frequently the difference between “ships inside the app” and “needs a server call for every keystroke.”

The second is large language model serving in the cloud, which most students encounter indirectly through tools like llama.cpp or the GGUF model file format that became popular for running large open models on ordinary consumer hardware. A 70-billion-parameter model stored in float16 needs roughly 140 GB of memory just for the weights, far beyond a single consumer GPU. Quantized down to 4-bit or 8-bit formats, the same model can shrink to a size that fits on a single high-end consumer GPU or even a capable laptop, which is exactly why quantized model formats are the default way enthusiasts run large models locally rather than the original full-precision weights.

In both cases the actual business reason for quantizing is rarely “we want our model to be slightly faster.” It is “we physically cannot fit the full-precision version where we need it to run, or it costs too much server memory to serve at our scale,” and quantization is the most direct lever for closing that gap.

There is also a less visible but equally real driver in cloud serving: GPU memory is often the actual bottleneck limiting how many requests a single server can handle at once, not raw compute. A smaller, quantized model leaves more GPU memory free for batching many users' requests together, which directly increases how many people one GPU can serve per second. This is a major reason production inference teams quantize models that would otherwise fit in full precision just fine: the win is throughput and cost per request at scale, not just fitting on a single device.

When Quantization Is Not Worth It

Quantization is not something to reach for automatically. A few situations where it is reasonable to skip it or be extra careful:

• The model is already small. If a model is a few megabytes and running on a server with plenty of RAM, the size reduction solves a problem you do not have, while still adding the accuracy risk.
• Your workload is compute-bound on hardware without int8 support. As shown above, without a genuine fast int8 path, you get the accuracy cost without the speed benefit, which is a bad trade.
• The domain is accuracy-critical in a way where even 1% matters. Medical diagnosis models, financial risk scoring used for real decisions, or anything with a regulatory accuracy requirement deserves careful, measured validation of the quantized model against real held-out data before shipping it, not just an assumption that “it's usually fine.”
• You have not benchmarked on the real target hardware. Given how wide the real-world speedup range turned out to be in the Android and Redis studies cited above (sometimes under 1x, meaning slower), shipping a quantized model to production without measuring actual latency on the actual device is a real risk, not a formality.

The right mental model is: quantization trades a small, measurable amount of accuracy for a guaranteed reduction in size and a conditional, hardware-dependent reduction in latency. Treat the size win as a given and the speed win as something you verify, not assume.

A reasonable default process, rather than treating this as a binary yes-or-no decision, is to quantize a copy of the model, run it against a held-out validation set that resembles real production inputs, and compare both the accuracy metric that matters for your product and the actual measured latency on your actual target hardware, before replacing the full-precision model anywhere in production. If either number moves more than your product can tolerate, that is useful information gained cheaply, well before a user ever notices.

Frequently Asked Questions