The Open-Weights Playbook: When to Self-Host a Model and When to Pay the API

• Also see: Creatine, magnesium, omega-3: what the evidence says
• Also see: Ghost streams over our heads
• Also see: The AI rogue panic

Two years ago this was not a real decision. You paid the API because open models were visibly worse. Today a 70B open-weight model handles summarization, classification, extraction, support triage, and retrieval-augmented generation about as well as last year’s flagship — at roughly a tenth of the per-token price if you run it yourself. The catch is the if: GPUs, engineering time, and use cases that actually fit.

This is a practical playbook for that choice, with the math teams actually use.

The landscape in one paragraph

On the open side you have dense small models like Mistral Nemo 12B that fit on a single workstation GPU, mid-size workhorses in the 70B class, and big mixture-of-experts models like NVIDIA Nemotron 3 Super 120B and GPT-OSS 120B that activate only a fraction of parameters per token and punch above their hosting cost. On the API side you have frontier systems — Claude Opus class, GPT-6 class, Gemini class — that still lead on hard reasoning, long-horizon agents, advanced code, and native multimodal work.

The gap has narrowed fastest on boring enterprise tasks. It remains widest on Olympiad math, competitive coding, and autonomous multi-step research. Most companies live in the first world and pay for the second.

flowchart TD
    A["Define workload and volume"] --> B["Check privacy and data rules"]
    B --> C["Estimate monthly token volume"]
    C --> D["Compare cost at scale"]
    D --> E["Test open model quality"]
    E --> F["Meets quality bar"]
    F -->|"yes"| G["Self host open weights"]
    F -->|"no"| H["Pay API per token"]
    G --> I["Size GPUs and monitor cost"]
    H --> J["Revisit as volume grows"]

Cost per 1M tokens with real numbers

API pricing moves constantly, but the shape is stable. Flagship reasoning APIs cost on the order of 15 dollars per blended 1M tokens once you mix input and output. Lightweight API minis cost around 1 dollar per 1M. A self-hosted 70B on rented H100s lands near 0.80 dollars per 1M at decent utilization. A self-hosted 12B lands near 0.20 dollars per 1M.

type: bar
title: Blended cost per 1M tokens in dollars
x: API flagship, API mini, Self-host 70B, Self-host 12B
Dollars: 15, 1, 0.8, 0.2

The chart understates the API advantage at low volume and the self-host advantage at high volume, because self-hosting is mostly fixed cost. An H100 at about 2 to 3 dollars per hour costs roughly 1500 to 2200 dollars per month. Two of them to serve a 70B at 16-bit run you 3000 to 4500 dollars per month before engineering time. At 10M tokens per month, that is ruinous versus API. At 500M to 1000M tokens per month, it wins by 5x to 10x.

Here is the break-even math teams use. Monthly self-host cost divided by API cost per token equals break-even tokens.

Take a 70B on 2xH100 at 4000 dollars per month all-in. Against a flagship API at 15 dollars per 1M, break-even is about 267M tokens per month. Against an API mini at 1 dollar per 1M, break-even is 4000M tokens — basically never, unless you drop to a smaller self-hosted model.

Take a 12B on 1xA100 or 1xH100 at 1500 dollars per month. Against an API mini at 1 dollar per 1M, break-even is 1500M tokens. That sounds high, but many classification and extraction pipelines burn billions of cheap tokens. Against flagship pricing, break-even drops to 100M tokens, which a single busy support bot can hit.

Utilization is everything. A GPU serving 5 requests per second around the clock produces roughly 10B tokens per month. Most internal tools do 5 percent utilization. If your traffic is spiky or bursty, APIs win because you pay zero for idle. If your traffic is a steady firehose — nightly document processing, continuous ingestion, always-on agents — self-hosting wins fast.

Introducing NVIDIA Nemotron 3 Super for multi-agent systemsPlays from YouTube when you press play

GPU sizing without guesswork

VRAM math is simple. Parameters times bytes per parameter equals base memory, plus 20 to 30 percent for KV cache and overhead.

A 12B dense model at 16-bit needs about 24 GB. That fits on a single RTX 4090, L40S, or A100 40GB. Quantize to 8-bit or 4-bit and it fits in 12 to 16 GB with minimal quality loss for most tasks. This is the self-host starter kit: one box, vLLM or TensorRT-LLM, done.

A 70B dense model at 16-bit needs about 140 GB. That is 2xH100 80GB with tensor parallelism, or 4xA100 40GB. Quantized to 8-bit it fits on 2xA100 40GB or 1xH100 with offload tricks, and INT4 builds fit on a single 80GB card for slower throughput. For production throughput — 50 to 100 tokens per second per user, dozens of concurrent users — budget 2xH100 and call it the sweet spot.

A 120B mixture-of-experts like Nemotron 3 Super 120B or GPT-OSS 120B is friendlier than it looks. Total parameters are 120B, but active parameters per token are far fewer, often 5B to 20B equivalent. At 16-bit the full weights need about 240 GB, so 4xH100 or 8xA100 for full precision. With 8-bit quantization the practical footprint drops to roughly 130 GB, fitting on 2xH100 80GB. With 4-bit it fits on a single 80GB card for experimentation. Throughput per dollar is excellent because decoding cost scales with active parameters, not total.

Add CPU RAM for offload, fast NVMe for weight staging, and do not forget networking if you split across nodes. Inference is memory-bandwidth bound. H100s win not just on capacity but on 3 terabytes per second of bandwidth.

When privacy forces the choice

Cost is negotiable. Data residency often is not. Healthcare, finance, legal, defense, and EU deployments frequently cannot send customer data to a third-party API, even with zero-retention promises.

Open weights solve this cleanly. The model runs in your VPC, your cloud region, or on metal you own. Logs stay local. Fine-tunes never leave. Auditors can inspect the exact checkpoint. Air-gapped deployments are possible with 12B to 70B quantized builds.

APIs have improved — zero data retention agreements, HIPAA-eligible endpoints, EU data boundaries — but if your policy says no external inference, that discussion is over. Self-host.

Latency points the same way for specific patterns. APIs add 300 to 1000 milliseconds of network and queueing overhead plus rate-limit variance. Self-hosted models on local GPUs deliver predictable time-to-first-token, crucial for voice agents, IDE autocomplete, and high-frequency trading-adjacent summarization. Conversely, frontier APIs with massive batched fleets often deliver higher sustained throughput than a two-GPU startup closet.

Customization is the open superpower

Fine-tuning is where open weights pull away. With the weights in hand you can do full fine-tuning, LoRA adapters, DPO preference tuning, continued pretraining on domain corpora, and logit-level control. A 70B fine-tuned on your tickets, your contracts, your runbooks will beat a generic flagship prompted with the same context, at lower serving cost.

API fine-tuning exists but is limited to whatever the provider exposes, with higher per-token hosting fees and no portability. If your moat is proprietary data — medical notes, legal precedent, factory logs — open weights let you bake it in.

The rule of thumb from production teams: if prompting a flagship gets you 80 percent and fine-tuning an open 70B gets you 92 percent on your eval, ship the fine-tune. If the task needs open-ended reasoning the open model cannot learn, stay on API.

When the API still wins

Pay the API when you need frontier reasoning, 1M-token context with reliable recall, native vision plus audio plus video, or tool-use agents that plan over dozens of steps. Nemotron 3 Super 120B is strong for multi-agent orchestration, but Claude Opus class and GPT-6 class still lead on the hardest chains.

Also pay the API when you ship in days not quarters. No GPUs to provision, no autoscaling to build, no midnight CUDA errors. Prototypes, hackathons, and low-volume features belong on API. Migrate when volume and evals justify it.

Many mature stacks end hybrid: open 12B for classification and guardrails at effectively zero marginal cost, open 70B for bulk RAG and summarization, flagship API for the 5 percent of queries that need real brains, routed by a small classifier. That router alone often cuts bills by 70 percent.

Self-host when volume is high and steady, privacy matters, or fine-tuning wins. Pay the API when you need the frontier, move fast, or traffic is spiky. Do the break-even math quarterly, because both sides get cheaper every six months — but open weights are catching up faster than APIs are discounting.

---

By OMMAIS: Muse Spark 1.3 using Cline’s cloud API