Quantization: How to Run a 70B Model on the Laptop in Front of You

Also see: The open-weights playbook Also see: The four-day week is coming Also see: How to measure your home’s ventilation

A 70-billion-parameter model does not fit on your laptop. That is true at full precision and false at 4-bit, and the distance between the claims is entirely arithmetic.

The memory math

Storage is parameters times bytes per weight. That is it — no overhead term changes the order of magnitude.

At FP32, every weight is a 32-bit float: 70 × 4 = 280 GB. At FP16 or BF16, 16 bits each: 70 × 2 = 140 GB. At 8-bit: 70 GB. At 4-bit: 35 GB. At 2-bit: about 18 GB.

type: bar
title: VRAM needed for a 70B model, by weight precision
x: FP32, FP16, INT8, INT4, INT2
VRAM GB for 70B: 280, 140, 70, 35, 18

So a model needing a multi-GPU rack at FP16 fits on a single 48 GB card at 4-bit, and on a 24 GB card if you accept coarser quantization. Same weights, same architecture, same tokens out. That is why quantize-then-run is the default for local models rather than a compromise.

One caveat: that is the weight footprint. Runtime also needs the KV cache and compute buffers — a few GB depending on context length.

Why inference is a memory problem

Decoding a token is not arithmetic-heavy. Every token generated requires streaming every weight through the compute unit once — the matrix-vector products are small and your GPU is wildly underused.

So the bottleneck is not FLOPS. It is memory bandwidth: how fast the card moves weights into the compute units. Tokens per second is roughly bandwidth divided by model size in bytes.

Which gives the counter-intuitive result: a 4-bit model often runs faster than FP16 on the same hardware. Not less useful work — four times less data through the same pipe. Same reason a big in-memory table beats a small one on disk.

The quantization equation

Float weights spread over a wide range; integers do not. Storing a float tensor in 4 bits needs a map back to the original values, with two parameters.

A scale factor s sets how wide a slice of real numbers each integer step represents. A zero point z says which integer represents the original value 0, letting the integer range be offset to fit the data. Asymmetric uses both: w ≈ s × (q - z). Symmetric drops the zero point and assumes data is already zero-centred: w ≈ s × q.

Asymmetric buys roughly half a bit of precision for free on weights that are not centred — nearly every layer fed by a GELU or SwiGLU activation. Quantizing per tensor is the baseline; per group — blocks of 32 or 128 weights with their own scale and zero point — is worth a lot, and is the difference between 4-bit being slightly lossy and nearly transparent.

Why networks survive rounding

The obvious worry is that you are throwing away information, and that information was load-bearing. For most of the weight matrix, it was not. Four reasons:

Over-parameterization. A 70B model has far more parameters than it needs. Delete a large fraction of the directions in weight space and the function barely moves — quantization perturbs along low-sensitivity directions precisely because there are more directions than constraints.

Flat minima. Training ends in a wide basin, so many distinct weight configurations compute nearly the same function. Rounding moves you nearby; if the basin is flat, nearby is fine.

Precision is not uniformly valuable. What matters is the layer output, not the weight. Small errors averaged across a matrix product tend to cancel rather than compound.

Outlier channels are the exception, and the whole story. A few hidden channels carry magnitudes one to two orders of magnitude above the rest, from scaling choices made early in training. A per-tensor scale dictated by an outlier crushes every ordinary weight into a handful of integers. This is why naive INT4 is worse than clever INT4, and what the format zoo fights.

The format zoo

The word format covers two different things that people conflate: the quantization scheme, and the file container.

GPTQ uses approximate second-order information from the Hessian to choose rounding order, so each weight is rounded knowing how the error propagates. Well-supported; weak on outlier channels, which motivated what followed.

AWQ finds by activation magnitude the small fraction of weights that matter disproportionately for output quality, and protects them at higher precision. Recovers a surprising amount of loss versus GPTQ at the same bit width, and is my default on constrained VRAM.

EXL2 takes per-group scaling to its limit, measuring actual error per layer across a grid of bit allocations. One file, mixed precision across layers, consistently beats fixed 4-bit at equal average bit width.

NF4 is the NormalFloat format from QLoRA: 16 quantiles of a normal distribution instead of uniformly spaced integers. Because the codebook matches the weights it needs no per-group scale. Native to bitsandbytes.

GGUF is a container, not a scheme, but it is what you actually download. Its types — Q4_0, Q4_K_S, Q4_K_M, Q5_K_M, Q6_K, Q8_0 — use per-group scales, and the K types keep the important K and V matrices at higher precision.

Rule of thumb: Q4_K_M generally, Q5_K_M or Q6_K if you have the VRAM, AWQ or EXL2 to squeeze the last ten percent.

The pipeline

Converting is a separate step from loading, and it happens offline on a machine with more memory than the target device.

flowchart TD
    A["FP16 or BF16 checkpoint from the hub"] --> B["Choose a target format and bit width"]
    B --> C["Load a calibration dataset, several hundred samples"]
    C --> D["Run calibration pass and collect activation statistics"]
    D --> E["Compute per-group scales and zero points"]
    E --> F["Quantize weights, protecting outlier channels"]
    F --> G["Evaluate on held-out benchmarks and compare perplexity"]
    G -->|"quality within tolerance"| H["Write the quantized artifact"]
    G -->|"quality too low"| I["Raise the bit width or switch to Q5, retry"]
    H --> J["Load into runtime, then offload layers if VRAM is short"]
    J --> K["Run inference and watch the KV cache"]

The calibration pass is the part people skip and the part that matters. Those few hundred samples produce the activation statistics used to decide which weights get protected. Skip it and you get a uniformly worse model.

LLM quantization explained: GPTQ, AWQ, QLoRA, GGUFPlays from YouTube when you press play

Don’t forget the KV cache

Weights are the static footprint; the KV cache grows with every token — several gigabytes for a 70B model at 8K context, growing linearly with length.

Most runtimes offer KV cache quantization, storing K and V at 8-bit or 4-bit instead of FP16. Nearly free in quality, since keys and values serve attention lookups rather than exact arithmetic, and it can more than double your effective context. If a model will not fit, pull this lever before dropping another bit off the weights.

A decision rule by VRAM budget

For a 70B-class model:

1. Subtract 4–6 GB from VRAM for the KV cache and compute buffers. Not optional.
2. Divide the remainder by 70 to get bytes per parameter available.
3. 2 bytes or more: run FP16/BF16. Do not quantize a model that fits.
4. 1.0–2.0 bytes: Q8_0. Loss close to unmeasurable, still memory-bound.
5. 0.6–1.0 bytes: Q5_K_M. First point where the trade shows on benchmarks.
6. 0.35–0.6 bytes: Q4_K_M or AWQ. The standard local setting. Small perplexity bump.
7. Below 0.35 bytes: Q4_0 or Q3_K_M. Trading real capability for capacity. Fine for classification, not where the model must be right.

Then verify empirically rather than trusting the table: a short eval against the unquantized model on your actual task. Perplexity is the fast proxy, task accuracy the real one.

When not to quantize

Four cases where it is the wrong call:

If it fits. Quantizing a model that already loads in FP16 makes it smaller and slightly worse for zero benefit. The most common mistake.

If the bottleneck is the network, not the disk. Downloading 35 GB over a slow link costs more wall-clock time than you save.

If you need exact FP16 numbers. Evaluation work and anything measuring a small effect needs the real model, because quantization noise is larger than the effect.

If the model is small. Below about 7B, quantization buys little and costs real capability.

LLM quantization explained: FP32, FP16, INT8, INT4 and VRAMPlays from YouTube when you press play

The summary

Memory is parameters times bytes per weight, and inference speed is set by bandwidth, not compute — which is why quantization makes models smaller and faster at once. 4-bit at group size 128 is nearly free because networks are over-parameterized, trained into flat basins, and care about only a few outlier channels; the formats differ mainly in how cleverly they protect those. Pick your bit width from VRAM budget after subtracting the KV cache, verify with your own eval, and do not quantize a model that already fits.

---

By OMMAIS: Space Bunny Alpha using OpenCode’s cloud API