Key summary
A learning rate is the value in gradient descent that determines how far to move the weights in a single step along the gradient direction. If it is too large, the loss jumps around or diverges; if it is too small, the loss decreases very slowly. Therefore, it is common to start with a moderate value and use a scheduler to reduce the learning rate as training progresses.
The learning rate is a core hyperparameter for understanding and fixing training runs where the loss jumps up and down or refuses to go down during gradient descent. This post walks through the update rule, how the loss behaves when the learning rate is too large or too small, how to pick a value through short experiments, and how learning rate schedulers and warmup work.
What is a learning rate?
One-line answer: A learning rate is a hyperparameter in gradient descent that scales how far a single update step moves the weights along the gradient direction.
In gradient descent, the general weight update rule is written as w ← w − lr × gradient. Here, lr (the learning rate) scales how far a single step moves the weights.
According to the PyTorch torch.optim documentation, optimizers take the learning rate via the lr argument—for example, when constructing optim.SGD(model.parameters(), lr=..., momentum=...) as shown in the documentation, or when using torch.optim.Adam, which also takes an lr argument. The gradients themselves come from backpropagation, which is covered in What is backpropagation? How a neural network uses gradients to fix its weights.
As an analogy, imagine walking down a foggy hill: the gradient tells you which way is downhill, while the learning rate determines how long your stride is.

What happens if the learning rate is too large or too small?
One-line answer: If the learning rate is too large, steps overshoot the bottom so the loss bounces up and down or diverges; if it is too small, the loss decreases very slowly or looks stuck.
Depending on the learning rate, the behavior of the training loss falls into the following qualitative cases:
- Too large: Each step overshoots the bottom of the valley, so the loss bounces up and down or grows (diverges).
- Too small: Each step moves only a tiny distance, so the loss decreases very slowly or looks stuck.
- About right: The loss decreases steadily without wild oscillations.

How do you choose a learning rate?
One-line answer: Because there is no single correct value for every case, try a few values in short experiments, compare the loss curves, and change one thing at a time.
There is no single universal learning rate that works for every model and dataset. Instead of launching a full training run immediately, try a few candidate values in short experiments and compare how their loss curves behave.
When tuning, change one thing at a time so you can tell what caused the difference. When the loss bounces up and down or grows, try a smaller learning rate; when the loss barely moves, try a larger learning rate.
What are learning rate schedulers and warmup?
One-line answer: A learning rate scheduler adjusts the learning rate during training, while warmup starts with a smaller learning rate for the initial steps and gradually raises it to the target value.
In the PyTorch documentation section “How to adjust learning rate”, torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs. Documented scheduler classes include StepLR, CosineAnnealingLR, LinearLR, and SequentialLR, while ReduceLROnPlateau allows dynamic learning rate reduction based on validation measurements.
Warmup is the general concept of starting with a smaller learning rate for the first steps and raising it gradually to the target value. For example, you can build such a schedule from the documented schedulers using LinearLR, or SequentialLR to chain phases together.
As documented in PyTorch, learning rate scheduling should be applied after the optimizer’s update—that is, call scheduler.step() after optimizer.step():
optimizer.step()
scheduler.step()
What are the frequently asked questions?
One-line answer: Here are answers on how batch size interacts with the learning rate, whether Adam still needs a learning rate, and how the learning rate is set during fine-tuning.
Are batch size and learning rate related?
Yes, they interact. Changing the batch size changes how noisy each gradient estimate is, so when you change the batch size you should re-check the learning rate.
If I use Adam, do I not need to set a learning rate?
No. torch.optim.Adam adapts per-parameter step sizes, but it still takes an lr argument that sets the learning rate.
Do you use the same learning rate during fine-tuning?
Generally it is set separately from pretraining, and is commonly chosen smaller to avoid wrecking what the model already learned. What fine-tuning is and how it differs from pretraining is covered in Pre-training, fine-tuning, and RLHF differences: What stages build an LLM?.
What related posts are worth reading?
One-line answer: You can check our related posts on language model training stages and AI basics study notes.
- Pre-training, fine-tuning, and RLHF differences: What stages build an LLM?
- AI basics study notes (Korean)
- What is backpropagation? How a neural network uses gradients to fix its weights
- What is overfitting? How to fix it when training loss falls but validation loss rises
What are the sources?
One-line answer: This post references the official PyTorch documentation on optimizers and learning rate scheduling.
- PyTorch Documentation,
torch.optim, https://docs.pytorch.org/docs/stable/optim.html (checked 2026-10-25) - PyTorch Documentation,
torch.optim.Adam, https://docs.pytorch.org/docs/stable/generated/torch.optim.Adam.html (checked 2026-10-25)