Key takeaway

Positional encoding is the information that tells a Transformer what position each token occupies. Because self-attention alone cannot distinguish token order, the original Transformer added sine and cosine values of different frequencies across dimensions to the embeddings. Instead of adding vectors, RoPE rotates the Query and Key vectors by their positions so that the dot product between two tokens depends only on their relative position (m − n).

Positional encoding is the mechanism that injects token position information into a Transformer so the model can tell word order apart. This post explains why Transformers need explicit order information, how sinusoidal positional encoding and learned positional embeddings work, and how Rotary Position Embedding (RoPE) differs by rotating Query and Key vectors.

Why is positional encoding needed?

One-line answer: Self-attention treats the input like an unordered set, so without injected position information, shuffling the words produces the same per-token results.

As Vaswani et al., “Attention Is All You Need” points out, the Transformer “contains no recurrence and no convolution”. For the model to make use of the order of the sequence, information about the relative or absolute position of the tokens in the sequence must be injected.

To see where positional encoding fits into the pipeline, it helps to connect it with the foundational components of a Transformer:

How is sinusoidal positional encoding built?

One-line answer: Each pair of dimensions uses a sine and cosine of a different frequency to form a position vector, which is then added directly to the token embedding.

In the original Transformer paper, the positional encodings have the same dimension d_model as the embeddings so that the two can be summed. For token position pos and dimension pair index i, the sinusoidal formulas are:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Here pos is the position and i is the dimension index. The wavelengths form a geometric progression from 2π to 10000·2π: low dimensions change rapidly as position increases, whereas high dimensions change very slowly. The authors hypothesized that this function would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PE(pos+k) can be represented as a linear function of PE(pos).

The following pe.py script and output show values checked in this post’s environment (Python 3.13.5, NumPy 2.5.4), using illustrative toy inputs (e.g., d_model=8, θ=1, q=(1,2), k=(3,1)):

import numpy as np
np.set_printoptions(precision=3, suppress=True)
d = 8
pos = np.arange(4)[:, None]
i = np.arange(d // 2)[None, :]
ang = pos / 10000 ** (2 * i / d)
pe = np.zeros((4, d)); pe[:, 0::2] = np.sin(ang); pe[:, 1::2] = np.cos(ang)
print("PE (pos 0-3, d_model=8)"); print(pe)
def rot(v, m, th=1.0):
    c, s = np.cos(m * th), np.sin(m * th)
    return np.array([c * v[0] - s * v[1], s * v[0] + c * v[1]])
q = np.array([1.0, 2.0]); k = np.array([3.0, 1.0])
for m, n in [(2, 1), (5, 4), (2, 0)]:
    print(f"m={m} n={n} m-n={m-n} q.k={rot(q, m) @ rot(k, n):.3f}")

Running python3 pe.py produces the following output:

PE (pos 0-3, d_model=8)
[[ 0.     1.     0.     1.     0.     1.     0.     1.   ]
 [ 0.841  0.54   0.1    0.995  0.01   1.     0.001  1.   ]
 [ 0.909 -0.416  0.199  0.98   0.02   1.     0.002  1.   ]
 [ 0.141 -0.99   0.296  0.955  0.03   1.     0.003  1.   ]]
m=2 n=1 m-n=1 q.k=-1.506
m=5 n=4 m-n=1 q.k=-1.506
m=2 n=0 m-n=2 q.k=-6.627

At position 0 (first row), alternating sine and cosine values give [0. 1. 0. 1. ...], and the later columns change much more slowly across positions than the first pair of columns.

Concept diagram: adding sinusoidal positional encoding vectors to token embeddings

How does it differ from learned positional embeddings?

One-line answer: Learned positional embeddings train a parameter vector for each position up to a maximum length, whereas sinusoidal encoding computes values from a fixed formula without trainable parameters.

Instead of a fixed trigonometric formula, learned positional embeddings assign a trainable vector to each position up to a configured maximum sequence length and optimize those vectors during training.

In the original Transformer experiments, learned positional embeddings and sinusoidal positional encoding produced “nearly identical results”. The authors chose the sinusoidal version because it “may allow the model to extrapolate to sequence lengths longer than the ones encountered during training”.

The table below compares learned absolute positional embeddings, sinusoidal positional encoding, and RoPE:

MethodHow values are determinedWhere position info is applied
Learned absoluteTrainedAdded to embedding
SinusoidalFormulaAdded to embedding
RoPEFormula (rotation)Multiplied (rotated) on Q and K

What is RoPE?

One-line answer: Rotary Position Embedding (RoPE) rotates the Query and Key vectors by an angle proportional to their position so that their dot product retains only the relative position (m − n).

As introduced in Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding”, RoPE “encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention”.

In the 2D case, rotating a projected vector at position m by angle mθ is written in complex form as:

f(x, m) = (W x)·e^(i m θ)

When Query q at position m is rotated by mθ and Key k at position n is rotated by nθ, the angle between them changes by (m − n)θ. Consequently, their dot product q_mᵀ k_n depends only on the relative distance m − n.

For a general dimension d, RoPE splits the d dimensions into d/2 pairs and rotates each pair i by angle m·θ_i:

θ_i = 10000^(−2(i−1)/d)

Unlike sinusoidal or learned embeddings, which are additive, RoPE is multiplicative (multiplying by an orthogonal rotation matrix). The last three lines of the pe.py output above illustrate this directly:

m=2 n=1 m-n=1 q.k=-1.506
m=5 n=4 m-n=1 q.k=-1.506
m=2 n=0 m-n=2 q.k=-6.627

Both (m=2, n=1) and (m=5, n=4) have the same relative distance m − n = 1 and yield the exact same dot product q.k = -1.506, whereas (m=2, n=0) has m − n = 2 and gives -6.627.

Concept diagram: rotating Query and Key vectors by position in RoPE

Why do many recent LLMs use RoPE?

One-line answer: It is explained by the fact that relative position appears directly in the attention dot product without adding vectors to the embedding, alongside active discussion around extending context length.

Why many recent LLMs adopt RoPE is generally explained through several design characteristics:

  • Relative position directly in the attention score: Rather than adding a position vector to the input embedding, RoPE is applied to Q and K so that relative position (m − n) appears directly in the attention dot product.
  • Preserving vector length: Because rotation uses an orthogonal matrix, it does not change the length (norm) of the vectors.
  • Properties noted in the RoFormer abstract: The RoFormer paper highlights flexibility of sequence length, decaying inter-token dependency with increasing relative distance, and the ability to equip linear self-attention with relative position encoding.
  • Active discussion on extending context length: There is active discussion around extending context length on top of RoPE.

That said, which positional encoding method works best depends on the model architecture and training setup; there is no single universal answer.

What are the frequently asked questions?

One-line answer: Here are quick answers on what happens without positional encoding, what ALiBi is, and why RoPE is not applied to the Value (V) vector.

What happens if positional encoding is removed?

Without positional information, self-attention alone cannot tell word order apart, so changing the word order (for example, an illustrative pair like “A dog bit a person” vs. “A person bit a dog”) the model could not tell the two orders apart, so the per-token attention results would not reflect the change in word order.

What is ALiBi?

Instead of adding position vectors to the embeddings, ALiBi adds a penalty proportional to the distance between tokens directly to the attention score.

Is RoPE also applied to V (Value)?

No. In the RoFormer formulation, rotation is applied to Q (Query) and K (Key), where the attention dot product is computed, while V (Value) carries the token content without rotation.

One-line answer: These posts cover self-attention, the Transformer structure, tokens and context windows, and embeddings that work alongside positional encoding.

What are the sources?

One-line answer: The sources are the original Transformer paper introducing sinusoidal positional encoding, the RoFormer paper introducing RoPE, and the code execution environment.