Quantization: How to Run a 70B Model on the Laptop in Front of You
A 70B model is 140 GB at FP16 and about 35 GB at 4-bit — the difference between an eight-GPU rack and one consumer card. How the math works, why it barely hurts quality, and how to pick a format.