Knowledge distillation is a technique for transferring knowledge from a large, cumbersome model or ensemble into a single smaller model that is much easier to deploy when running the large model directly for many users is too expensive.

Key summary

Knowledge distillation trains a smaller student model by using the output probability distribution of a large, well-trained teacher model as targets. Rather than learning only the single correct answer, the student also learns how similar different answers are to one another. While quantization lowers the numerical precision of the same model, distillation trains a separate, smaller model.

What is knowledge distillation?

One-line answer: It is a method that trains a small, easy-to-deploy student model to match the class probability distributions produced by a large teacher model or ensemble.

A cumbersome model or an ensemble of multiple models can be too expensive to deploy to a large number of users. Knowledge distillation transfers the knowledge of that large model (the teacher) into a single smaller model (the student) that is practical to deploy. To do this, the small model is trained to match the class probabilities produced by the large model as “soft targets”, either on the original training set or on a separate transfer set.

The student learns from the teacher's probabilities

Why use soft targets instead of hard labels?

One-line answer: Soft targets preserve the relative probabilities assigned to wrong answers, carrying much more information per training case about how the teacher generalizes than one-hot hard labels.

A trained teacher model assigns probabilities to all wrong answers as well, and the relative sizes of those probabilities reveal how the model generalizes. In the paper’s example, an image of a car may have only a very small chance of being mistaken for a garbage truck, yet that probability is still much higher than mistaking it for a carrot. Soft targets with high entropy therefore provide far more information per training case than hard one-hot labels.

To produce a suitably soft probability distribution, the temperature of the final softmax is raised in the teacher model. The small student model is trained using the same high temperature to match those soft targets, and once training is finished, the student uses a normal softmax temperature of 1 at inference.

When the correct labels are known, training works better by taking a weighted average of two objectives: cross-entropy with the soft targets at a high temperature and cross-entropy with the correct labels at normal temperature (1).

Both distillation temperature and sampling temperature divide logits before the softmax, but distillation temperature is a training-time knob to soften targets whereas sampling temperature is a generation-time knob for randomness (see “What are temperature and top-p?”).

How does it differ from quantization?

One-line answer: Quantization stores and computes the same model with fewer bits, whereas knowledge distillation trains a different, smaller model.

As covered in What is quantization? Differences between FP16, INT8, and INT4 and why LLM memory shrinks (Korean), quantization keeps the same model and represents or computes its numbers with fewer bits. Knowledge distillation instead trains a separate, smaller student model. The two methods can also be combined—for example, a distilled model can be quantized afterward.

Quantization and distillation change different things

When is knowledge distillation used?

One-line answer: It is used when deploying models to phones or edge devices, or when serving requests where latency and inference cost must be kept low.

Knowledge distillation is useful when a model needs to run on phones or edge devices with constrained resources, or in latency- and cost-sensitive serving environments. For which hardware accelerator runs models on edge devices, see What is the difference between an NPU and a GPU? (Korean).

What are the frequently asked questions?

One-line answer: Here are answers regarding student model capability, Mixture of Experts (MoE), and fine-tuning.

Can the student model outperform the teacher?

There is no general guarantee that a student model will beat its teacher; usually the goal is to get close to the teacher’s capability at a lower deployment cost.

Is it related to Mixture of Experts (MoE)?

They are different ideas: MoE routes inputs to experts inside one model, whereas distillation transfers knowledge into a separate model. See What is Mixture of Experts (MoE)? Differences from dense models and memory overhead for details.

How does it differ from fine-tuning?

Fine-tuning keeps training the same model on new data, whereas distillation trains a new smaller model to imitate a teacher’s outputs. See Pre-training, fine-tuning, and RLHF: How is an LLM built? for a comparison of training stages.

One-line answer: These related articles cover quantization, MoE, LLM training stages, edge accelerators, and sampling temperature.

What are the sources?

One-line answer: This post is based on the paper that introduced knowledge distillation, soft targets, and temperature.

  • Geoffrey Hinton, Oriol Vinyals, Jeff Dean, “Distilling the Knowledge in a Neural Network”, arXiv:1503.02531 (submitted 2015-03-09), https://arxiv.org/abs/1503.02531 (checked 2026-10-25)