Summary

An LLM inference framework is the runtime that reads trained model weights, accepts requests and produces tokens. It helps to split engines into server-style ones (such as vLLM), which batch many requests to use GPUs efficiently, and local-style ones (such as llama.cpp), which run quantized models lightly on laptops, CPUs and small GPUs.

One-line answer: Split engines into server-style (vLLM and similar) and local-style (llama.cpp and similar), then choose by number of users, hardware and model format.

Inference engines built for a specific model family also became a big topic on Hacker News recently. This post explains the concepts, so it includes no install or run commands.

What is an LLM inference framework, and why isn’t a model file enough?

One-line answer: A weights file is only data, so you need an engine that handles loading, tokenizing, the token-by-token generation loop and the API server.

A model file holds only trained weights. Separate software has to load them into memory, turn input text into tokens with a tokenizer, run the loop that generates one token after another, and return results to the caller. That is the job of an inference framework. The vLLM docs list streaming outputs, an OpenAI-compatible API server and Hugging Face model integration, while the llama.cpp README describes GGUF model files and a built-in OpenAI-compatible API server.

Flow from request through tokenization and token generation to response in an LLM inference framework

If tokens and context are new to you, see our token and context guide first.

How do server-style and local-style engines differ?

One-line answer: Server-style engines batch concurrent requests to use GPUs efficiently; local-style engines run quantized models lightly on CPUs and small GPUs.

AspectServer-style (e.g. vLLM)Local-style (e.g. llama.cpp)
Main goalProcess many incoming requests togetherRun lightly on modest resources
Memory and batchingPagedAttention, continuous batchingCPU+GPU hybrid inference (for models larger than VRAM)
QuantizationFP8, INT8, INT4, GPTQ/AWQ, GGUF and more1.5-bit to 8-bit integer quantization
HardwareDistributed inference with tensor/pipeline parallelismDependency-free C/C++, Apple silicon first-class (NEON, Accelerate, Metal)
Model formatHugging Face model integrationGGUF files
APIOpenAI-compatible server, streaming outputsIncludes an OpenAI-compatible server

We covered quantization in the quantization guide. KV cache is not re-explained here; see AI Day3: LLM inference and KV cache if you are curious.

How should you choose one?

One-line answer: Decide by concurrent users, hardware, model format and whether you need multiple GPUs.

  • Number of users: Serving many users at once points to server-style; just you points to local-style.
  • Hardware: With server GPUs, go server-style; on a laptop, CPU or small GPU, local-style fits.
  • Model format: safetensors/Hugging Face format suits server-style; GGUF files suit local-style.
  • OpenAI-compatible API: Both offer one, so it is not a deciding factor.
  • Multi-GPU: If you must split a model across GPUs, look at server-style engines with tensor/pipeline parallelism.

Choosing server-style vs local-style engines by concurrent users, hardware, model format, and multi-GPU needs

Why do model makers release their own inference engines these days?

One-line answer: Most likely to build in optimizations tuned to a specific model architecture. This is general reasoning, not an official explanation.

On 2026-10-20, a Hacker News post titled “DeepInfer: High-performance inference framework for DeepSeek models” drew attention to an inference engine built for a specific model family (HN thread). The general idea is that optimizing for one model architecture can include tricks a general-purpose engine cannot. That is only general reasoning, not a claim based on the thread’s content or figures.

What are the frequently asked questions?

Can several people use a local-style engine?

It is possible, but batching concurrent requests is the core strength of server-style engines. If several people need it at once, server-style is the better fit.

Does quantization reduce quality?

It can, and it depends on the model and method. See the quantization guide for details.

Why does an OpenAI-compatible API matter?

Because you can reuse existing client code almost as is. Both vLLM and llama.cpp provide an OpenAI-compatible API server.

What are the sources?

One-line answer: The vLLM docs, the llama.cpp GitHub README and a Hacker News thread, all checked on 2026-10-21.

External sources