Local and Air-Gapped LLMs, What Actually Works Offline
Pull-quote: “Offline is not a degraded mode. It is a different architecture with its own bill of materials, and the model is the easiest item on the list.”
Why this matters
Some data cannot leave the building. Defense networks, sovereign-cloud mandates, manufacturing floors with proprietary process data, health records under residency law: for these environments, “just call the API” is not a policy discussion; it is a non-starter. The good news is that fully offline LLM systems are now genuinely viable. The catch is that the model is the easiest part. What separates working air-gapped deployments from stalled ones is everything around the model: serving, retrieval, and evaluation, all of which silently assume the internet unless you engineer otherwise.
The full offline bill of materials
Air-gapped enclave
───────────────────────────────────────────────────────
Serving Retrieval Evaluation
──────── ───────── ──────────
vLLM (prod, Local embedding ◄── Golden set +
batched, GPU) model regression suite
│ │ │
Ollama (dev, Local vector Local judge model
workstation) store + BM25 (calibrated vs.
│ │ human sample)
▼ ▼ ▼
Quantized Local reranker Human review for
open-weight high-stakes calls
models ─── every arrow stays inside the wall ───Serving. The division of labor is settled: vLLM for production, where continuous batching and paged attention turn one GPU into a multi-tenant service, and Ollama for development, where an engineer needs a model on a workstation in minutes. Teams that ship Ollama to production discover the throughput ceiling; teams that force vLLM onto laptops discover the operational overhead. Use both, for what each is for.
Quantization. Weight quantization is the lever that fits real models onto procurable hardware; 4-bit weights need roughly a quarter of the memory of 16-bit. The tradeoff is real but uneven:
| Precision | Memory vs FP16 | Quality behavior |
|---|---|---|
| FP16 / BF16 | 1× | Reference quality |
| 8-bit | ~½ | Generally indistinguishable in practice |
| 4-bit | ~¼ | Fine for most tasks; degrades first on long-chain reasoning and math |
| Below 4-bit | <¼ | Task-specific gamble; evaluate before trusting |
The rule: quantization damage is task-dependent, so the only trustworthy answer comes from your own evaluation set run at each precision, not from a leaderboard measured on someone else’s tasks.
Model selection. For on-prem you are choosing among open-weight families, and the checklist is short: a license compatible with commercial deployment, a context length that fits your retrieval design, tool-calling support if agents are in scope, and, above all, verified performance on your tasks. Model releases outpace any static recommendation; the durable asset is the evaluation harness that lets you re-decide quickly.
Retrieval. The forgotten line items. A RAG stack needs an embedding model and usually a reranker, and both must also run inside the wall. Local embedding models and cross-encoder rerankers are mature; pair dense vectors with BM25 hybrid retrieval, which costs nothing extra offline and hedges against embedding blind spots.
Evaluation without cloud
No cloud also means no frontier-model judge. Working offline evaluation rests on three legs: a golden set of labeled task examples that functions as the regression suite for every model, quantization, and prompt change; a local judge model for scaled scoring, calibrated against a human-scored sample before its scores are trusted; and mandatory human review for high-stakes outputs. None of this is a degraded copy of the cloud workflow. A system designed to run with either cloud or local models proves the deeper point: the architecture, not the deployment target, decides what is possible, and the evaluation harness is the part of the architecture that travels.
Closing
Air-gapped LLM systems fail on the parts nobody budgeted: the embedding model that assumed an API, the reranker nobody procured, the evaluation loop that quietly required a frontier judge. Inventory the full bill of materials, serve with vLLM in production and Ollama at the desk, quantize as far as your own eval set tolerates, and build the golden-set harness first, because offline, it is the only source of truth you have.