Summary
An LLM inference framework is the runtime that reads trained model weights, accepts requests and produces tokens. It helps to split engines into server-style ones (such as vLLM), which batch many requests to use GPUs efficiently, and local-style ones (such as llama.cpp), which run quantized models lightly on laptops, CPUs and small GPUs.
One-line answer: Split engines into server-style (vLLM and similar) and local-style (llama.cpp and similar), then choose by number of users, hardware and model format.
Inference engines built for a specific model family also became a big topic on Hacker News recently. This post explains the concepts, so it includes no install or run commands.
What is an LLM inference framework, and why isn’t a model file enough?
One-line answer: A weights file is only data, so you need an engine that handles loading, tokenizing, the token-by-token generation loop and the API server.
A model file holds only trained weights. Separate software has to load them into memory, turn input text into tokens with a tokenizer, run the loop that generates one token after another, and return results to the caller. That is the job of an inference framework. The vLLM docs list streaming outputs, an OpenAI-compatible API server and Hugging Face model integration, while the llama.cpp README describes GGUF model files and a built-in OpenAI-compatible API server.

If tokens and context are new to you, see our token and context guide first.
How do server-style and local-style engines differ?
One-line answer: Server-style engines batch concurrent requests to use GPUs efficiently; local-style engines run quantized models lightly on CPUs and small GPUs.
| Aspect | Server-style (e.g. vLLM) | Local-style (e.g. llama.cpp) |
|---|---|---|
| Main goal | Process many incoming requests together | Run lightly on modest resources |
| Memory and batching | PagedAttention, continuous batching | CPU+GPU hybrid inference (for models larger than VRAM) |
| Quantization | FP8, INT8, INT4, GPTQ/AWQ, GGUF and more | 1.5-bit to 8-bit integer quantization |
| Hardware | Distributed inference with tensor/pipeline parallelism | Dependency-free C/C++, Apple silicon first-class (NEON, Accelerate, Metal) |
| Model format | Hugging Face model integration | GGUF files |
| API | OpenAI-compatible server, streaming outputs | Includes an OpenAI-compatible server |
We covered quantization in the quantization guide. KV cache is not re-explained here; see AI Day3: LLM inference and KV cache if you are curious.
How should you choose one?
One-line answer: Decide by concurrent users, hardware, model format and whether you need multiple GPUs.
- Number of users: Serving many users at once points to server-style; just you points to local-style.
- Hardware: With server GPUs, go server-style; on a laptop, CPU or small GPU, local-style fits.
- Model format: safetensors/Hugging Face format suits server-style; GGUF files suit local-style.
- OpenAI-compatible API: Both offer one, so it is not a deciding factor.
- Multi-GPU: If you must split a model across GPUs, look at server-style engines with tensor/pipeline parallelism.

Why do model makers release their own inference engines these days?
One-line answer: Most likely to build in optimizations tuned to a specific model architecture. This is general reasoning, not an official explanation.
On 2026-10-20, a Hacker News post titled “DeepInfer: High-performance inference framework for DeepSeek models” drew attention to an inference engine built for a specific model family (HN thread). The general idea is that optimizing for one model architecture can include tricks a general-purpose engine cannot. That is only general reasoning, not a claim based on the thread’s content or figures.
What are the frequently asked questions?
Can several people use a local-style engine?
It is possible, but batching concurrent requests is the core strength of server-style engines. If several people need it at once, server-style is the better fit.
Does quantization reduce quality?
It can, and it depends on the model and method. See the quantization guide for details.
Why does an OpenAI-compatible API matter?
Because you can reuse existing client code almost as is. Both vLLM and llama.cpp provide an OpenAI-compatible API server.
What are the sources?
One-line answer: The vLLM docs, the llama.cpp GitHub README and a Hacker News thread, all checked on 2026-10-21.
Related posts
External sources
- vLLM docs (checked 2026-10-21)
- llama.cpp GitHub (checked 2026-10-21)
- Hacker News: DeepInfer (posted 2026-10-20, checked 2026-10-21)