An optimizer is the rule in neural network training that decides how to update each weight using the computed gradients. Even with the same gradients from the loss, the training trajectory changes depending on whether you step directly opposite the current gradient, add inertia from previous steps, or scale step sizes per parameter. This post explains how gradient-descent optimizers—SGD, momentum, Adam, and AdamW—differ and how to pick a starting point.
Summary
An optimizer takes the gradients computed by backpropagation and decides what direction and how far to move each weight. SGD steps opposite the gradient with the same step size scale, momentum adds the previous moving direction like inertia to reduce zigzagging, and Adam adapts each parameter’s step size separately using running averages of gradients and squared gradients. Adam or AdamW is a common starting default, and whichever optimizer you choose, you still need to set a learning rate.
What is an optimizer?
One-line answer: An optimizer is the algorithm that takes the loss gradients computed by backpropagation and decides the direction and step size for updating every weight.
Neural network training separates computing gradients from applying weight updates. Backpropagation computes the gradient of the loss for every weight (see What is backpropagation? How a neural network uses gradients to fix its weights), and the optimizer decides how to use those gradients to update the weights—both the direction and the step size.
How does SGD move?
One-line answer: Stochastic gradient descent (SGD) moves every weight in the direction opposite the mini-batch gradient by the same step size scale (the learning rate).
SGD looks only at the gradient from the current mini-batch and takes a step directly opposite that gradient, scaling the step for all weights by a shared learning rate.
When the loss surface forms a long, narrow valley, the gradient points much more steeply across the walls than along the floor. As a result, SGD tends to zigzag back and forth across the steep walls while moving slowly along the valley floor.
What does momentum add?
One-line answer: Momentum keeps a running direction from previous steps and adds it like inertia, so zigzag components cancel out and movement along a consistent direction speeds up.
Momentum works like a ball rolling downhill and building up inertia. Instead of relying solely on the current step’s gradient, it maintains a running direction from previous steps and combines it with the new gradient.
Across a narrow valley, the side-to-side components that flip sign at every step cancel each other out, while the components pointing steadily along the valley floor accumulate and accelerate progress. In PyTorch’s torch.optim, this is enabled by passing a momentum argument to the SGD optimizer class.
How is Adam different?
One-line answer: Adam keeps running averages of both the gradient and the squared gradient per parameter, using them to adapt each parameter’s step size separately.
Introduced by Kingma and Ba, Adam tracks two running averages for every parameter: the running average of the gradient (the first moment, capturing direction like momentum) and the running average of the squared gradient (the second moment, capturing how large the gradients have been). Using both moments, Adam adapts the step size for each parameter individually rather than applying a uniform scale across all weights.

In PyTorch’s torch.optim, Adam and AdamW are separate optimizer classes. According to the PyTorch AdamW documentation (referencing the paper Decoupled Weight Decay Regularization), AdamW implements decoupled weight decay, “where weight decay does not accumulate in the momentum nor variance.”
Which one should you pick first?
One-line answer: Adam or AdamW is a common default starting point, while SGD with momentum remains widely used for many models such as vision architectures.
No single optimizer is always best for every problem, but the general trade-offs and common starting points can be summarized as follows:
| Optimizer | Key characteristics | Commonly used when |
|---|---|---|
| SGD + momentum | Adds a running direction from past steps (inertia) to the mini-batch gradient with a simpler update rule. | Widely used across many vision models and tasks where a simple, well-understood update rule works well. |
| Adam | Adapts each parameter’s step size separately using running averages of the gradient and squared gradient. | Frequently used as a default starting point when setting up a new training run. |
| AdamW | Keeps Adam’s per-parameter adaptive step sizes while decoupling weight decay from the momentum and variance terms. | A common default starting point, especially for transformer-style models. |
Whichever optimizer you pick, you still have to choose a learning rate.
What are the frequently asked questions?
One-line answer: You still have to choose a learning rate regardless of the optimizer, and neither preventing overfitting nor defining the loss is the optimizer’s job.
If I pick an optimizer like Adam, do I still need to choose a learning rate?
Yes. Even when an optimizer adapts step sizes per parameter, you still have to choose the learning rate; see “What is a learning rate? How to adjust it when the loss jumps around or won’t go down” for how the learning rate is adjusted.
Does the optimizer prevent overfitting?
No. The optimizer’s role is to update weights to reduce the training loss, not to prevent overfitting; for ways to handle rising validation loss, see What is overfitting? How to fix it when training loss goes down but validation loss goes up.
How is an optimizer different from a loss function?
The loss function defines what error the model is trying to minimize, whereas the optimizer defines how to move the weights using gradients to reduce that loss. For an example of a classification loss function, see “What is cross-entropy loss? Why classification models use it instead of MSE”.
What related posts are worth reading?
One-line answer: These related posts cover how backpropagation computes gradients, how to handle overfitting, and how language models are trained across stages.
- What is backpropagation? How a neural network uses gradients to fix its weights
- What is overfitting? How to fix it when training loss goes down but validation loss goes up
- Pre-training vs fine-tuning vs RLHF: What stages build an LLM?
What are the sources?
One-line answer: This post is based on the official PyTorch documentation and the original Adam paper.
- PyTorch Documentation — torch.optim (checked 2026-10-28)
- PyTorch Documentation — torch.optim.AdamW (checked 2026-10-28)
- Diederik P. Kingma, Jimmy Ba — Adam: A Method for Stochastic Optimization (checked 2026-10-28)