Key summary
An activation function adds nonlinearity to the output of each layer so a neural network can learn complex relationships that a single straight line cannot represent. Without an activation function, stacking any number of layers collapses into a single linear transformation. Sigmoid squashes outputs into (0, 1), ReLU clips negative values to 0, and GELU transitions smoothly around 0.
An activation function sits after each linear layer in a neural network and introduces nonlinearity into the layer’s output. This post explains mathematically why neural networks need activation functions, how sigmoid, ReLU, and GELU differ in definition and shape, how they connect to the vanishing gradient problem, and how to choose an activation for hidden layers versus the output layer.
What is an activation function and why is it needed?
One-line answer: Putting a nonlinear function between linear layers keeps stacked layers from collapsing into a single linear map and allows the network to represent bent or curved decision boundaries.
A single linear layer computes Wx + b for an input x. If you stack two linear layers back to back without a nonlinearity in between, the combined operation simplifies algebraically:
W2(W1x + b1) + b2 = (W2W1)x + (W2b1 + b2)
Because the result is still one linear map with weight matrix W2W1 and bias W2b1 + b2, a network built only from linear layers can only draw straight decision boundaries no matter how many layers are stacked. Placing a nonlinear activation function between layers lets the network represent bent and curved boundaries.

How do sigmoid, ReLU, and GELU differ?
One-line answer: Sigmoid squashes any input into (0, 1), ReLU outputs 0 for negative inputs and the identity for positive inputs, and GELU weights inputs smoothly around 0 by their value rather than gating strictly by sign.
According to the official PyTorch documentation (torch.nn, Non-linear Activations, checked at version 2.14), the core functions are defined as follows:
ReLU(x) = max(0, x)
Sigmoid(x) = 1 / (1 + exp(-x))
GELU(x) = x * Φ(x)
Tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x))
Softmax(x_i) = exp(x_i) / Σ_j exp(x_j)
In GELU(x) = x * Φ(x), Φ(x) is the cumulative distribution function (CDF) of the standard Gaussian distribution. PyTorch’s torch.nn.GELU also offers a tanh-based approximation via the approximate='tanh' option. As Hendrycks and Gimpel describe in Gaussian Error Linear Units (GELUs) (arXiv:1606.08415, submitted 2016-06-27), GELU weights inputs by their value rather than gating them by their sign as ReLU does.
From these definitions, the three functions take distinct shapes:
- Sigmoid: squashes any real input into the interval
(0, 1)and flattens at both ends. - ReLU: outputs
0for negative inputs and acts as the identity function (x) for positive inputs. - GELU: stays close to
0for very negative inputs, stays close toxfor large positive inputs, and bends smoothly around0(dipping slightly below0for some negative inputs).
| Function | Output range | Shape | Common placement |
|---|---|---|---|
| Sigmoid | (0, 1) | S-shaped curve that flattens at both ends. | Output layer for a single yes/no probability |
| ReLU | [0, ∞) | 0 for negative inputs and identity (x) for positive inputs. | Hidden layers |
| GELU | From slightly below 0, unbounded above | Close to 0 for very negative inputs, close to x for large positive inputs, smooth around 0. | Hidden layers |

How do activation functions relate to the vanishing gradient problem?
One-line answer: Sigmoid’s slope gets close to 0 at both flat ends, so multiplying many such slopes across deep layers during backpropagation can make gradients near the input side very small, whereas ReLU keeps a constant slope for positive inputs.
Because sigmoid flattens at both ends, its slope approaches 0 when the input magnitude is large. When many layers use sigmoid and their slopes are multiplied together during backpropagation, the gradients reaching layers near the input side can become very small—a phenomenon known as the vanishing gradient problem.
ReLU, by contrast, has a constant slope for positive inputs, which is one reason it is widely used in hidden layers. How gradients are propagated backward through a network is covered in What is backpropagation? How a neural network uses gradients to fix its weights.
How do you choose an activation function?
One-line answer: Match the output layer activation to the task target—sigmoid for a single yes/no probability, softmax for choosing one of several classes, or often no activation for predicting a real-valued number—and commonly use ReLU-family functions or GELU in hidden layers.
Choosing an activation function is easiest when you separate the output layer from the hidden layers:
- Output layer: depends on the form of prediction the task requires. Use sigmoid for a single yes/no probability, softmax for choosing one of several classes, and often no activation when predicting a real-valued number.
- Hidden layers: commonly use ReLU-family activations or GELU to provide nonlinearity between linear layers.
If training does not progress well even with a suitable activation function, it is also worth checking the learning rate alongside the activation choice, as covered in the post “What is a learning rate? How to adjust it when the loss jumps around or won’t go down”.
What are the frequently asked questions?
One-line answer: Here are concise answers on dead ReLU units and LeakyReLU, how PyTorch classifies softmax, and when tanh is used.
What is a dead ReLU?
By definition, ReLU has both output and slope equal to 0 for negative inputs. If a unit’s input stays negative across all data, it receives no gradient and stops updating, which is called a dead ReLU. Variants such as LeakyReLU (available in torch.nn) keep a small slope for negative inputs.
Is softmax also an activation function?
PyTorch lists Softmax among its activation modules (“Non-linear Activations (other)”), and it is typically used at the output layer to rescale a vector of scores so the elements lie in [0, 1] and sum to 1.
When is tanh used?
tanh squashes inputs into (-1, 1) and is centered at 0. It is useful when zero-centered outputs are desired, though like sigmoid it flattens at both ends.
What related posts are worth reading?
One-line answer: These posts cover language model training stages and AI basics, and companion posts cover backpropagation and the learning rate, which work directly alongside activation functions during training.
- Pre-training, fine-tuning, and RLHF differences: What stages build an LLM?
- AI basics study notes (Korean)
- What is backpropagation? How a neural network uses gradients to fix its weights
- “What is a learning rate? How to adjust it when the loss jumps around or won’t go down”
What are the sources?
One-line answer: This post is grounded in the official PyTorch torch.nn documentation and the GELU paper.
- PyTorch Documentation,
torch.nn— Non-linear Activations (torch.nn.ReLU,torch.nn.GELU,torch.nn.Sigmoid,torch.nn.Tanh,torch.nn.Softmax, version 2.14), https://docs.pytorch.org/docs/stable/nn.html (checked 2026-10-25) - Dan Hendrycks, Kevin Gimpel, “Gaussian Error Linear Units (GELUs)”, arXiv:1606.08415 (submitted 2016-06-27), https://arxiv.org/abs/1606.08415 (checked 2026-10-25)