Key summary

Overfitting means the model fits the training data so closely that it fails on new data. If validation loss starts rising while training loss keeps falling, suspect it. Common things to try are early stopping, more data, regularization and dropout, or a smaller model.

When you train a model, there comes a moment when the two lines on the loss graph pull apart. Training loss keeps improving while validation loss turns upward at some point, and that is overfitting. This post walks through how to read the loss curve, how to diagnose the problem, and the order in which to fix it.

What is overfitting?

One-line answer: Overfitting is when the model matches the training set so closely that it predicts new data poorly.

Think of it as memorizing answers instead of understanding the material. Memorized questions get answered correctly, but a slightly different question trips you up. A model that has learned the underlying pattern works on data it has never seen, and that ability is called generalization: performing well on new data. In bias–variance terms, overfitting is a model leaning toward high variance.

How do you read a loss curve?

One-line answer: Training loss and validation loss fall together at first, and the point where validation loss turns upward while training loss keeps falling is where overfitting begins.

A loss curve plots loss against training iterations. It is usually read in phases.

  • Both losses falling together: the model is learning general patterns in the data.
  • The gap widening: only training loss keeps dropping faster, and the model starts fitting the training data alone.
  • Validation loss rising: training loss keeps falling while validation loss goes up. Around this point is where early stopping comes in.

Loss curve and early stopping

How do you diagnose overfitting?

One-line answer: Compare training and validation loss. A large gap with low training loss means overfitting, and both being high means underfitting.

Start by checking the gap between training loss and validation loss. If training loss is low but validation loss is much higher, the model is overfitting. If both are high, the model has not learned the pattern yet, which is underfitting, and the remedy is different. When validation results look suspiciously good, also check for data leakage, because results gained through leakage are not generalization.

  • Check that validation data has not been mixed into the training data.
  • Check that there are no duplicates across the splits.
  • Check that no input feature gives away the answer.

Overfitting vs underfitting diagnosis

How do you fix overfitting?

One-line answer: Try these in order: early stopping, more or augmented data, L2 regularization and dropout, then a smaller model.

Early stopping

Early stopping ends training at or near the point where validation loss stops improving. It needs almost no changes to your setup, so it is the first thing to try.

More data and augmentation

More and more diverse data keeps the model from latching onto accidental patterns. If collecting more data is hard, you can also consider data augmentation, a widely used method that creates more training data by transforming the data you already have.

L2 regularization and dropout

Regularization penalizes model complexity, and L2 regularization is the common form. Dropout is another widely used method: it randomly drops units during training.

A smaller model

A simpler model with fewer parameters can sometimes generalize better. If the steps above are not enough, try reducing the model size.

What are the frequently asked questions?

One-line answer: The questions below are answered in terms of principles.

Is overfitting always bad?

Overfitting is a signal that the model fails on new data, so it is a problem to fix. A small gap between training loss and validation loss is normal, though, and is not overfitting by itself.

How much data should I hold out for validation?

Principles matter more than a fixed ratio. The validation set must never be used for training, must resemble the real data, and must be large enough that its score is stable. Keep a separate test set for the final check.

Can fine-tuning overfit too?

Yes. Fine-tuning on a small dataset can overfit. For the difference between pretraining and fine-tuning, see this guide to pretraining, fine-tuning and RLHF.

One-line answer: Reading the basics and model architecture alongside this post makes it easier to follow.

What are the sources?

One-line answer: This post is based on Google’s Machine Learning Crash Course.