Cross-entropy loss (also called log loss) is the standard loss function for evaluating how much probability a classification model assigns to the true class. This post explains how the loss changes with the predicted probability, how it connects to softmax, and why classification models use cross-entropy instead of mean squared error (MSE).
Key summary
Cross-entropy loss penalizes a classification model more heavily the lower the probability it assigns to the true class, defined as -log(p) for the true class probability p. As p approaches 1, the loss approaches 0; as p approaches 0, the loss grows without bound. Classification models use it instead of mean squared error (MSE) because, when paired with softmax probabilities, it produces a large gradient when the model is confidently wrong, allowing weights to update quickly.
What is cross-entropy loss?
One-line answer: It is a loss function defined as the negative natural log of the predicted probability p for the correct class (-log(p)), which penalizes the model steeply as p approaches zero.
For a single example where the model assigns probability p to the correct class (for example, the true class cat among cat, dog, and bird), cross-entropy loss is defined using the natural logarithm as -log(p). As p approaches 1, the loss approaches 0; as p approaches 0, the loss grows without bound.
The table below shows the exact values computed with Python’s math.log (natural log, rounded to three decimal places). For instance, when the correct class probability is 0.9, the loss is about 0.105, whereas if the probability drops to 0.01, the loss rises to about 4.605.
Correct Class Probability (p) | Cross-Entropy Loss (-log(p)) |
|---|---|
0.9 | 0.105 |
0.5 | 0.693 |
0.1 | 2.303 |
0.01 | 4.605 |

In information theory, cross-entropy and Kullback-Leibler (KL) divergence measure how much a predicted probability distribution diverges from a target distribution, and in machine learning this loss is also known as log loss. When the target is a one-hot distribution where only the true class has probability 1, the calculation simplifies directly to -log(p).
How does it connect to softmax?
One-line answer: Softmax converts the model’s raw scores (logits) into probabilities summing to 1, and cross-entropy picks only the probability of the true class to compute -log(p).
In a multi-class classifier that distinguishes between cat, dog, and bird, the pipeline works as follows:
- The final layer outputs unnormalized raw scores (logits) for each class (cat, dog, bird).
- The
softmaxfunction converts these logits into positive probabilities summing to 1. - Only the probability
pof the correct class (such as cat) is selected, and-log(p)is computed as the loss.
For how softmax turns scores into normalized weights, see What is Self-Attention? How Q, K, and V weight tokens, and for how hidden-layer nonlinearities compare, see What is an activation function? ReLU vs GELU vs sigmoid, and how to choose.
In practice, when using PyTorch’s torch.nn.CrossEntropyLoss, pass raw logits directly and do not apply softmax yourself beforehand. According to the PyTorch documentation, this criterion computes the cross-entropy loss between input logits and target, expecting unnormalized logits for each class (which do not need to be positive or sum to 1, in general), and with class-index targets it is equivalent to applying LogSoftmax on an input followed by NLLLoss.

Why use cross-entropy instead of MSE for classification?
One-line answer: When a model is confidently wrong, cross-entropy produces a large penalty and a large gradient for fast weight updates, whereas MSE on saturated softmax outputs can yield a small gradient that slows learning down.
When a classifier produces softmax probabilities, cross-entropy and mean squared error (MSE) behave very differently during training:
- Cross-entropy: When the model is confidently wrong (so the true class probability
pis near 0), cross-entropy gives a large penalty and a large gradient, so learning moves fast where it should. - Mean squared error (MSE): MSE’s penalty on a probability is bounded, and its gradient can become small when the output is saturated, so a confidently wrong model learns slowly.
How gradients update network weights is covered in What is backpropagation? How a neural network uses gradients to fix its weights, while tuning the step size is the topic of What is a learning rate? How to adjust it when the loss jumps around or won’t go down. Meanwhile, MSE remains the natural choice for regression tasks where the model predicts a continuous value.
How is it used in LLM training?
One-line answer: Because next-token prediction in a large language model is classification over the whole vocabulary, pretraining optimizes cross-entropy loss on the next token.
In large language models (LLMs), next-token prediction is framed as multi-class classification over the entire vocabulary. At each step, the model produces logits for every token in the vocabulary, and pretraining uses cross-entropy loss on the true next token.
How this pretraining objective fits into the broader LLM pipeline is explained in Pretraining vs Fine-Tuning vs RLHF: What Stages Build an LLM?. In addition, perplexity—a common evaluation metric for language models—is defined as the exponential of the average cross-entropy loss.
What are the frequently asked questions?
One-line answer: Here are concise answers on binary classification loss, whether zero loss is desirable, and how label smoothing works.
What is used for binary classification?
For binary classification problems that distinguish between true and false, models use binary cross-entropy (BCE) loss.
Is it good when the loss reaches zero?
Not necessarily. Driving training loss close to zero can mean the model has memorized the training data and loses performance on new data due to overfitting; see What is overfitting? How to fix it when training loss falls but validation loss rises.
What is label smoothing?
Label smoothing mixes the target distribution with a uniform distribution to prevent overconfident predictions, and PyTorch’s torch.nn.CrossEntropyLoss includes a label_smoothing parameter. For soft labels in general, see What is knowledge distillation? Training a small model from a big one, and how it differs from quantization.
What related posts are worth reading?
One-line answer: These related posts cover backpropagation, overfitting, Self-Attention, LLM training stages, and related training fundamentals.
- What is backpropagation? How a neural network uses gradients to fix its weights
- What is overfitting? How to fix it when training loss falls but validation loss rises
- What is Self-Attention? How Q, K, and V weight tokens
- Pretraining vs Fine-Tuning vs RLHF: What Stages Build an LLM?
- What is a learning rate? How to adjust it when the loss jumps around or won’t go down
- What is knowledge distillation? Training a small model from a big one, and how it differs from quantization
- What is an activation function? ReLU vs GELU vs sigmoid, and how to choose
What are the sources?
One-line answer: Below are the official PyTorch documentation and machine learning reference materials checked for this post.
- PyTorch Documentation — torch.nn.CrossEntropyLoss (checked 2026-10-28)
- Google Machine Learning Crash Course (checked 2026-10-28)