RoPE: giving language models a sense of position

Language models like ChatGPT, Gemini or Claude don’t read text word by word, one after another, the way humans do. They read the whole text at once, in parallel. This brings many advantages, such as making full use of GPUs, but it also raises a problem: how does the model know the order of the words? Let’s explore the solution most widely adopted by modern models: RoPE.

Why position matters

In modern language models (LLMs), based on the transformer architecture, the first step is to preprocess the text, turning it into a numerical representation that is better suited to a deep learning model. This process has two parts: tokenization, which splits the text into fragments (tokens), and embedding, which creates vector representations of those tokens (embeddings).

Text

“In 2014 a paper changed everything.”

tokenization

Tokens

In

2014

a

pap

er

chang

ed

every

thing

.

embedding

Embeddings

0.32-0.710.140.88⋮

-0.540.910.23-0.47⋮

0.670.12-0.390.55⋮

0.81-0.220.76-0.09⋮

0.440.19-0.520.37⋮

-0.330.570.91-0.64⋮

0.71-0.280.490.15⋮

0.29-0.850.410.16⋮

0.620.08-0.270.44⋮

0.05-0.110.030.08⋮

Pieces of the same word

An embedding, on its own, captures the meaning of a token. However, the meaning of a text doesn’t depend only on the meaning of each token, but also on their order. “The dog chases the cat” and “The cat chases the dog” contain exactly the same tokens, but they don’t mean the same thing.

The

dog

chases

the

cat

The dog is the one chasing.

The

cat

chases

the

dog

The cat is the one chasing.

no position

chases

cat

The

dog

the

What the model sees

By default, a transformer LLM receives a sequence of embeddings and processes them in parallel, as an unordered set. Here “in parallel” doesn’t mean that tokens are processed in isolation: the model uses a mechanism called “attention”. At every layer, each token looks at the ones that come before it, decides how much each of them weighs, and reinterprets itself based on that mixture.

To do this, each token’s embedding is projected into three vectors by three learned matrices, and it works much like a database: the query is what the token is looking for, the key is what it offers to the others, and the value is the information it contributes. Each token compares its query with the key of every previous token (and with its own) using a dot product, and the result is the attention score, one for each pair of tokens. Those scores are normalized and decide how much each token’s value weighs in the new representation of the token that asked the question. In other words, queries and keys decide who talks to whom, and values are what gets said.

key

key

key

key

query

The

dog

chases

the

cat

q · k

1.2

2.6

3.4

1.8

3.0

weight

5%

18%

41%

8%

28%

1 · Compare query · key of each token → score

2 · Normalize scores → weights that add up to 100%

3 · Mix sum of weight × value → new “cat”

This mechanism is where the lack of order hurts: mixing a token with the ones around it without knowing which come before or after is treating the text as a bag of words (setting aside the fact that each token can only look backwards). That’s why we need some way to introduce the notion of order.

Two questions

The history of positional encoding is usually told as a timeline: first APE, then ALiBi, then RoPE, each one better than the last. But it’s much easier to understand if, instead of a list, we see it as a design space with two independent questions:

What is encoded: absolute position or relative distance? This question determines how the model behaves with sequences longer than the ones seen in training. An absolute position that has never been seen is an unknown value, whereas a relative distance of 3 tokens is the same at position 10 as at position 10,000.

At what point in the process is that information injected? This question determines how much order interferes with semantics. Modifying the input embedding is not the same as intervening only in the attention computation.

APE, ALiBi and RoPE are different answers to this pair of questions.

Partial answers

APE: absolute and in the embedding

The first solution, introduced in Attention Is All You Need itself, was Absolute Position Encoding (APE). Before the model processes each token’s embedding, a fixed vector is computed for every position in the sequence and added directly to the embedding (there are variants in which this vector is learned during training, but it works the same way). This way the embedding carries both semantic and positional information.

Embedding of each token

Enpos. 1

2014pos. 2

unpos. 3

…pos. n

+

+

+

+

Fixed position vector

PE(1)

PE(2)

PE(3)

PE(n)

Final embedding: meaning + position

En + PE(1)

2014 + PE(2)

un + PE(3)

… + PE(n)

An obvious solution would have been to number the tokens (1, 2, 3…), but those numbers grow without bound and have a very different scale from the embeddings, whose values are usually between −1 and 1: adding them would distort the semantic information. That’s why sine (even dimensions) and cosine (odd dimensions) are used. They always produce values in that range and, being 90° out of phase, together they unambiguously describe any point on a rotation.

Going back to our two questions, APE answers absolute to the first and in the input embedding to the second. And that is where its two limitations come from.

The first, a consequence of encoding absolute positions, is that the model doesn’t generalize well to sequences longer than those seen in training. If we train the model on sequences of 512 tokens, the positional vectors for positions 513, 514 or 515 are patterns it has never seen and doesn’t know how to interpret. Moreover, relative distance (which is usually what matters in language) is not explicitly encoded: the model has to reconstruct it from two absolute positions, and it finds it easier to know that a token is at position 5 than to know that it is 3 positions away from another.

The second, a consequence of acting on the input embedding, is that once the positional vector is added, positional and semantic information live in the same space and become indistinguishable for the following layers. The model has to devote capacity to separating the two signals, and that contamination is carried through the whole network.

ALiBi: relative and on the attention

Another relevant solution was ALiBi (Attention with Linear Biases), contemporary with RoPE. Its proposal was to leave the embeddings untouched and penalize the attention between two tokens according to the distance that separates them: the farther apart they are, the less attention they pay to each other.

q·k score with token n+1

Enpos. 1

2014pos. 2

⋯

todopos. n

+

+

+

+

Distance-based penalty

−nn tokens away

−(n−1)n−1 tokens away

⋯

−11 token away

Resulting attention: higher the closer it is

low

a bit higher

⋯

highest

ALiBi answers relative to the first question and on the attention to the second, and with that it fixes both problems of APE at once. The embeddings stay clean, and the penalty is a formula that can be evaluated for any distance, whether or not the model has seen it: if it has never seen a 600-token sequence, it simply applies it.

The price is a different limitation: the penalty is not learned, it is a fixed bias that the model cannot turn off, and it always reduces attention with distance. ALiBi imposes as a rule something that in language is only true on average: that what is near matters more than what is far. A proper name mentioned on the first page of a long document is still relevant a hundred thousand tokens later.

Besides, the penalty is added to an attention score that has already been computed and depends only on distance, without looking at what the tokens say. Position corrects the comparison between contents, but it doesn’t take part in it.

RoPE: relative and inside the attention

RoPE (Rotary Position Embedding) is the positional-information encoding mechanism introduced in the RoFormer paper (2021) and later adopted by most modern LLMs.

Here it’s worth clearing up a common misunderstanding, because RoPE is often described as “APE but rotating instead of adding”. The operation changes, but what matters is where it happens: RoPE doesn’t touch the input embedding. It acts inside each attention layer, rotating the query and key vectors right before computing their dot product. The value vectors and the rest of the token’s representation stay intact. That is, position influences who each token talks to, but not what that token contains.

RoPE is based on a fundamental but simple property of geometry: the dot product between two vectors rotated in the same plane, apart from the content of the two vectors, depends only on the difference between their rotation angles, not on the absolute angles. This is very useful, because the product between query and key is what the model uses to decide how much attention two tokens pay each other. Since each token is rotated by an amount proportional to its position, the angle difference between two tokens is precisely their relative distance. Position comes in as absolute and comes out as relative, with no need to explicitly compute the distance between every pair of tokens.

Token nposition n → rotates θ·n

Token mposition m → rotates θ·m

θ·(n−m)the angle between them depends only on the distance

The diagram is only a simplification of what RoPE does. In practice there is a key difference: RoPE doesn’t rotate the whole vector, but splits its dimensions into pairs and applies $d/2$ 2D rotations instead of a single $d$-dimensional rotation. This allows each pair to rotate at a different frequency, so that they capture distances at different scales (and, incidentally, 2D rotations are much cheaper to compute).

The mechanism is very similar to a clock, except that instead of measuring time it measures distance in a sequence of tokens. Each pair of dimensions, with its own rotation frequency, represents a hand of this clock. High-frequency pairs (which rotate fast) capture short distances, while low-frequency pairs (which rotate slowly) capture long distances. Together, they describe the distance between tokens at all scales at once, just as the three hands of a clock determine the exact time.

8-dimensional query/key vector, in pairs

x₀

x₁

Pair 0θ₀ largerotates fast

x₂

x₃

Pair 1θ₁ mediumrotates at a medium pace

x₄

x₅

Pair 2θ₂ smallrotates slowly

x₆

x₇

Pair 3θ₃ very smallrotates very slowly

Same token, same position: each pair has rotated by a different angle.

The RoPE formula for a pair of dimensions $(x_1, x_2)$ at position $m$ is:

$$\begin{pmatrix} x'_1 \\ x'_2 \end{pmatrix} = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$$

And the rotation frequency of each pair:

$$\theta_i = 10000^{-2i/d}$$

Where:

• $x_1, x_2$: the two dimensions of the query or key vector that form the pair
• $m$: position of the token in the sequence
• $\theta_i$: rotation frequency of pair i (decreases with i)
• $i$: index of the pair (0, 1, 2, …)
• $d$: dimension of the query or key vector
• $10000$: base constant that controls the range of frequencies

With this, RoPE fills the slot that none of the previous solutions covered: it is relative, like ALiBi, but it acts inside the query-key product rather than on the result of that product. And that distinction is what matters. A bias adjusts a comparison that has already been made, without knowing what the compared tokens contained. RoPE, on the other hand, changes the comparison itself: distance enters the computation together with content, and what comes out is a score in which both have taken part at once.

With one caveat: RoPE isn’t entirely neutral either. As each pair rotates at a different rate, their contributions drift out of alignment with distance and attention tends to decay. But it isn’t an imposed rule: it is irregular rather than monotonic, and the model can compensate for it.

Summing up the two questions from the beginning:

Method	What it encodes	Where it acts
APE	Absolute position	On the input embedding
ALiBi	Relative distance	On the already computed attention score (fixed bias)
RoPE	Relative distance	Inside the computation of the attention score

What RoPE doesn’t solve

It is often said that RoPE solves extrapolation, but that isn’t true. A model trained with RoPE at 4,096 tokens doesn’t work well at 16,000 either: the formula can be evaluated at any position, but the low-frequency pairs barely complete a fraction of a turn during training, and beyond that they enter angles they have never seen. The model can compute the rotation, but it doesn’t know what it means. That is why techniques like Position Interpolation or YaRN exist, which extend the context window of already trained models by rescaling those angles. They work because the whole position lives in an angle, and an angle can be rescaled without touching anything else.

ALiBi, on the other hand, does reach longer sequences, but it arrives empty: since the penalty grows without bound with distance, distant tokens end up muted and attention degenerates into a sliding window. It gains length, not context. RoPE fails more honestly: it breaks exactly where what it has seen runs out, and that is why it can be fixed.

What explains its adoption, then, is not extrapolation, but the place where it acts: RoPE doesn’t correct a comparison that has already been made, but makes distance part of it, with a negligible added cost and without muddying what each token means. It is no coincidence that models like LLaMA, Gemma or Qwen have adopted it as a standard, and that it is most likely also behind models like GPT or Claude, whose technical details are not public. When a model understands that “3 tokens away” means the same at position 10 as at position 10,000, it is RoPE turning the hands.

References

1. Vaswani et al., Attention Is All You Need (2017)
2. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
3. Press, Smith and Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021)
4. Chen et al., Extending Context Window of Large Language Models via Positional Interpolation (2023)
5. Peng et al., YaRN: Efficient Context Window Extension of Large Language Models (2023)