Local LLMs: Private AI on Your Own Hardware

Every prompt you send to a hosted API is a business decision. For most workloads that decision is fine. For the rest (medical records, legal discovery, source code, anything under NDA), the right answer is a model running on hardware you control, and in 2026 that answer is finally good enough to ship.

Why local is finally practical

Two things changed. First, open weight models got genuinely good. The current crop of 7B to 70B parameter models handles summarization, extraction, classification, retrieval augmented generation, and code assistance at a level that covers most enterprise workloads. Second, the tooling matured. What required a research team in 2023 is a Docker command today.

The case for local inference is not just privacy, although privacy is usually the trigger:

• Data never leaves your network. No third party processor, no cross border transfer, no vendor terms of service to re-read every quarter. For GDPR, HIPAA, and Tunisian data protection contexts, this collapses an entire category of compliance work.
• Predictable unit economics. API pricing scales linearly with tokens forever. A GPU server is a fixed cost. At sustained high volume, the crossover point arrives faster than most teams expect. We covered the API side of this math in LLM cost optimization.
• Latency you control. No rate limits, no shared capacity, no degraded service during someone else's launch day.
• Offline and air gapped operation. Factories, ships, clinics, defense adjacent environments. If the workload cannot reach the internet, the model has to live inside.
• No silent model changes. A pinned local model behaves the same on day 400 as on day 1. Hosted models get updated under you.

The honest counterargument: frontier hosted models are still smarter. If your workload needs the absolute ceiling of reasoning ability, local is not there yet. Most workloads do not need the ceiling.

The stack, from weights to endpoint

A production local LLM deployment has three layers: the model, the quantization format, and the inference server. Get any one wrong and the whole thing underperforms.

Choosing the model

Ignore leaderboard drama. Choose by workload class:

• 7B to 9B class: classification, extraction, routing, simple summarization. Cheap, fast, runs on a single consumer GPU.
• 20B to 35B class: the sweet spot. Strong RAG answers, solid instruction following, usable code generation. This is where most of our deployments land.
• 70B and above: multi step reasoning, hard synthesis tasks. Real hardware money, real ops burden. Deploy only when evals prove the smaller tier fails.

Two non-obvious checks. Read the license, actually read it: several popular "open" models carry commercial restrictions, user count thresholds, or field of use clauses that surface at the worst possible time. And prefer instruct tuned variants with safetensors weights. Legacy pickle based checkpoint files can execute arbitrary code on load, which is a supply chain problem, not a theoretical one.

Quantization: the compromise that makes it work

Model weights dominate memory. A 70B model at FP16 needs roughly 140 GB just for weights, before you serve a single token. Quantization stores weights at lower precision, typically 4 to 8 bits, cutting memory by 2x to 4x with a modest quality cost.

The formats that matter:

• GGUF (llama.cpp ecosystem): runs on CPU, Apple Silicon, and GPUs, with flexible offloading. The default for edge and workstation deployments.
• AWQ / GPTQ: GPU native 4-bit formats, well supported by production servers.
• FP8: on recent NVIDIA hardware, near lossless at half the memory of FP16. Use it when the silicon allows.

Rules of thumb from our deployments: 4-bit is the floor for production quality on most models, quality degrades sharply below it, and smaller models suffer more from quantization than larger ones. Never trust vibes. Run your eval set on the full precision model, then on the quantized artifact, and compare numbers.

Budget memory for the KV cache too. Long contexts at high concurrency can eat as much VRAM as the weights themselves. Total memory is roughly weights plus (KV cache per token, times context length, times concurrent requests). This is the line item that surprises everyone.

The inference server

Three tiers, three purposes:

• Ollama: developer laptops, prototypes, single user tools. Excellent onboarding, not a production server.
• llama.cpp / llama-server: CPU heavy or mixed hardware, GGUF models, embedded scenarios.
• vLLM (or SGLang, TGI): production. Continuous batching, paged KV cache management, tensor parallelism across GPUs, and an OpenAI compatible API.

Continuous batching is the feature that separates toys from infrastructure. Naive servers process requests one at a time, so ten concurrent users mean ten times the latency. vLLM interleaves requests at the token level and keeps the GPU saturated. In our experience this is the difference between serving 3 users and serving 50 on the same card.

A realistic production launch:

docker run --gpus all -p 8000:8000 \
  -v /srv/models:/models \
  vllm/vllm-openai:latest \
  --model /models/qwen2.5-32b-instruct-awq \
  --quantization awq \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --api-key "$LLM_GATEWAY_TOKEN"

Because the API is OpenAI compatible, application code barely changes:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.internal.example.com/v1",
    api_key=os.environ["LLM_GATEWAY_TOKEN"],
)

That one property, endpoint compatibility, is what makes hybrid architectures cheap to build later.

Hardware that actually works

Sizing is where budgets go to die, so here is the honest map:

• Single 24 GB GPU (RTX 4090 or L4 class): 7B to 14B models at 4-bit with comfortable context. The workhorse tier for internal tools.
• 48 to 96 GB (RTX 6000 Ada, L40S, or paired consumer cards): 32B class models served properly, or 70B quantized with tight context budgets.
• Multi GPU servers (4x to 8x data center cards): 70B and above at production concurrency using tensor parallelism.
• Apple Silicon: unified memory makes a Mac Studio a superb development box for surprisingly large models. It is not a concurrency machine. Prototype on it, serve elsewhere.
• CPU only: viable for batch jobs and background pipelines where nobody watches a spinner. Wrong for anything interactive.

The classic sizing mistake is benchmarking with one user. Every concurrent request adds its own KV cache, and throughput per user drops as batches deepen. Load test with your real prompt lengths, your real output lengths, and your real peak concurrency before you sign a hardware invoice.

Failure modes we see in the wild

Every one of these comes from a real engagement:

• The demo to production cliff. Works beautifully for the one engineer testing it, collapses at ten concurrent users because the server has no batching and no queue. Symptom: p99 latency measured in minutes.
• Silent quantization regressions. The 4-bit model ships because it "seems fine," then extraction accuracy quietly drops on the exact field the business cares about. Only an eval suite catches this.
• Context overflow truncation. The RAG pipeline stuffs 20k tokens into a 16k window, the server truncates from the top, and the system prompt vanishes. Answers get weird and nobody knows why. Fail loudly on overflow, never truncate silently.
• The unauthenticated port 8000. An inference endpoint on the office network with no auth is an open proxy to your most sensitive data flows. Local does not mean trusted.
• Unmonitored GPUs. VRAM fragmentation, a driver update from an unattended upgrade, thermal throttling in a closet "server room" in a Sousse summer. Without GPU metrics you find out from angry users.
• Prompt injection still applies. A local model processing untrusted documents can still be steered into leaking context or misusing tools. The threat model shrinks, it does not disappear.

Security is not a side effect of being local

Moving inference on premise removes the third party data processor. It does not remove security engineering, it relocates it to your team:

• Put the endpoint behind a gateway with token auth and rate limits, and segment the GPU network from general office traffic. The principles in our zero trust architecture guide apply directly: the model server is a workload, not a trusted zone.
• Treat prompt and completion logs as sensitive data, because they are. They now contain the exact material you went local to protect. Encrypt, retention limit, access control.
• Verify model provenance. Pull weights from official sources, pin checksums, prefer safetensors. A poisoned model file is a backdoor with a friendly filename.
• Red team the application layer: injection, context leakage, tool misuse. Same discipline as any API surface.

Local, API, or both: the decision framework

Ask four questions, in order:

1. Can the data legally and contractually leave your infrastructure? If no, the decision is made. Local.
2. Does a 30B class open model pass your eval set? If no, and only a frontier model passes, you need the API (possibly with data minimization or redaction in front of it).
3. Is volume sustained and high? Bursty low volume favors API economics. Steady high volume favors owned hardware.
4. Do you have, or will you buy, the ops capacity? A local deployment is a production service with GPUs. Someone must own it.

Most real answers are hybrid. Route by sensitivity and difficulty: the local model handles the routine, private, high volume 80 percent, and a hosted frontier model handles hard reasoning over non-sensitive inputs. Because everything speaks the OpenAI protocol, the router is a small piece of gateway code, not a rewrite.

RAG deserves a mention here, because it is the pattern that makes mid size local models punch above their weight. A 32B model with excellent retrieval routinely beats a much larger model with none. If that pipeline is new to you, start with RAG systems explained.

Your first deployment, step by step

1. Build an eval set first. 50 to 100 real examples with expected outputs, from your actual workload. This is the single highest leverage artifact in the whole project.
2. Shortlist two or three models in the smallest plausible size class, and verify each license against your commercial reality.
3. Baseline against a frontier API on the eval set, so you know the gap you are accepting (or not accepting).
4. Quantize and re-run evals. Ship the quantized model only if the numbers hold.
5. Size hardware from measured load, not from a blog post's rule of thumb, including peak concurrency and context length.
6. Deploy vLLM behind an authenticated gateway with rate limiting and structured logging.
7. Load test with production shaped traffic. Watch time to first token and p99, not averages.
8. Wire monitoring: GPU memory and temperature, queue depth, tokens per second, error rates. Alert before users do.
9. Ship behind a feature flag with API fallback, so a model or hardware failure degrades gracefully instead of taking the feature down.
10. Re-evaluate quarterly. Open model quality moves fast. The model you rejected in January may win in June, and swapping is cheap when your evals and gateway already exist.

That last point is the quiet superpower of this architecture. Once evals, serving, and routing are in place, upgrading your AI is a config change, not a project.

How Innovation T can help

Innovation T designs and ships private AI systems end to end: model selection and evaluation, quantization and serving infrastructure, RAG pipelines, GPU sizing, and the security hardening that makes an internal endpoint production grade. We have done this on cloud GPUs, on premise servers, and fully air gapped environments, and we are blunt about when a hosted API is simply the better call.

If your data is the reason you have not shipped AI yet, that is exactly the problem we solve. See our software and cloud engineering services or talk to our team about a scoped pilot.