RoPE: RoFormer, Enhanced Transformer with Rotary Position Embedding

Rotate each query and key by an angle set by its position, and the attention score depends on position only through the distance between the two tokens.

RoFormer replaces the position vector that a Transformer adds to each token with a rotation of the query and key inside attention. The rotation has no learned parameters, and LLaMA, Mistral, PaLM and GPT-NeoX all use it.

The construction rests on one identity: rotating a vector by mθ and another by nθ, then taking their dot product, gives the same number as rotating only the second by (n − m)θ. The page builds RoPE from that identity, runs it on an eight-number example, follows the paper's decay bound and its linear-attention variant, and ends with how Position Interpolation, NTK-aware scaling and YaRN change the rotation speeds to run past the training length.

Attention ignores word order

Self-attention weighs every pair of tokens by a dot product. For a query at position $m$ and a key at position $n$, with head width $d$, the weight is

and the output $\mathbf{o}_m$ is a weighted average of the value vectors. In the original Transformer the query and key are linear projections of the token embeddings, $\mathbf{q}_m = \mathbf{W}_q\mathbf{x}_m$ and $\mathbf{k}_n = \mathbf{W}_k\mathbf{x}_n$, and an embedding is a function of the word alone. Move “mat” from position 6 to position 2 and its key does not change, so its score against “sat” does not change. Permute the whole sentence and every output is permuted the same way: self-attention is permutation-equivariant. “The cat sat on the mat” and “the mat sat on the cat” give the same outputs, attached to different slots.

Figure 1 shows the weights from the query “sat” to five words. Cycle the word order and watch the heights and the corner readout for “mat”: they travel with the words and no number changes.

Every Transformer therefore injects position somewhere. Before RoFormer the options came in two families, which the paper surveys in its Section 2: absolute encodings add a vector for position $m$ to the token at $m$, and relative encodings change each pair's score according to the offset between the two tokens.

Position encodings before RoPE

The original Transformer adds a fixed sinusoidal vector $\mathbf{p}_m$ to each embedding: coordinate pair $t$ of $\mathbf{p}_m$ is the sine and cosine of $m/10000^{2t/d}$, so each pair oscillates at its own frequency. BERT and GPT-2 instead learn $\mathbf{p}_m$ as a table with one row per position up to a maximum length, and a learned table has no row for any position past that maximum. Either way the query becomes $\mathbf{W}_q(\mathbf{x}_m + \mathbf{p}_m)$, and the score expands into four terms (the paper's Eq. 6):

The first term compares content with content. The second and third compare one token's content with the other token's absolute position, and the fourth compares the two absolute positions through the learned matrix $\mathbf{W}_q^\top\mathbf{W}_k$. Nothing in (2) makes these terms depend on $m$ and $n$ only through their difference, so the same two words three tokens apart can score differently at positions (3, 6) and at (103, 106), and the model has to learn from its training positions alone that the offset is what matters.

Relative encodings edit (2) directly. Shaw, Uszkoreit and Vaswani (2018) add to the key a learned vector for the clipped distance $\mathrm{clip}(m-n, r_{\min}, r_{\max})$. Transformer-XL replaces $\mathbf{p}_n$ by a sinusoidal encoding of $m-n$ and the query-side position terms by two learned vectors. T5 drops the position terms and adds one learned scalar per relative distance bucket, $b_{m,n}$, to the content score. Each of these is a term computed for a particular $(m, n)$ pair, and RoFormer's introduction notes that because they “add the position information to the context representation” they are “unsuitable for the linear self-attention architecture”. The linear-attention section below shows why.

RoFormer asks for the relative property up front. Find functions $f_q$ and $f_k$ that put position into the query and key such that their inner product depends on the two embeddings and the offset only (the paper's Eq. 11):

If (3) holds, a pair of tokens at a given offset gets the same score at position 5 and at position 50,000, and attention needs no extra term per pair.

How RoPE works: rotate the query and key

Start with $d = 2$, so the projected query $\mathbf{q} = \mathbf{W}_q\mathbf{x}_m$ and key $\mathbf{k} = \mathbf{W}_k\mathbf{x}_n$ are arrows in a plane. RoPE rotates the query by the angle $m\theta$ and the key by $n\theta$, for a fixed rate $\theta$:

A rotation keeps lengths, and its transpose is the rotation by the opposite angle, so $\mathbf{R}_{m\theta}^\top\mathbf{R}_{n\theta} = \mathbf{R}_{(n-m)\theta}$. The score is therefore

and both absolute positions have dropped out: the right side depends on the content vectors and on $n-m$. In geometric terms, a dot product is $|\mathbf{q}||\mathbf{k}|$ times the cosine of the angle between the arrows. Rotating the query by $m\theta$ and the key by $n\theta$ changes that angle by $(n-m)\theta$, so the score is $|\mathbf{q}||\mathbf{k}|\cos((n-m)\theta + \varphi)$, where $\varphi$ is the angle from $\mathbf{q}$ to $\mathbf{k}$ before any rotation.

A numerical check with $\theta = 0.1$ and $\mathbf{q} = \mathbf{k} = (1, 0)$: at $m = 5$, $n = 7$ the query points at 0.5 rad and the key at 0.7 rad, and $(0.878, 0.479)\cdot(0.765, 0.644) = 0.980$, which is $\cos 0.2$. At $m = 105$, $n = 107$ the arrows point at 10.5 and 10.7 rad, both different from before, and the dot product is again $\cos 0.2 = 0.980$.

The paper states the same solution with complex numbers (its Eq. 12). Read the two coordinates of a 2D vector as the real and imaginary parts of one complex number; rotating by $m\theta$ is then multiplication by $e^{\mathrm{i}m\theta}$, and the dot product of two vectors $a$ and $b$ is $\mathrm{Re}[a\,b^*]$:

The paper writes the offset as $m-n$ here and as $n-m$ in its matrix form; the two differ by the sign of $\theta$, and this page uses $n-m$ throughout. Figure 2 draws the 2D case at $\theta = 0.4$, with unit vectors so that the score is the cosine alone.

Why a rotation and not some other position-dependent map? Section 3.4.1 of the paper derives it. Write $f_q$, $f_k$ and $g$ each as a length and an angle. Setting $n = m$ in (3), and requiring that position 0 leaves a vector unchanged, gives two conditions: the product of the query and key lengths at any common position equals $|\mathbf{q}||\mathbf{k}|$, and the difference of their angles equals the difference before encoding. The paper takes the “straightforward solution” for the lengths, which is to leave them alone, and finds the angle must be the content angle plus a term $\phi(m)$ shared by queries and keys. Setting $n = m + 1$ then shows that $\phi(m+1) - \phi(m)$ is the same for every $m$, so $\phi(m) = m\theta + \gamma$, and the paper sets $\gamma = 0$. A map that keeps the length and adds an angle growing linearly with position is a rotation by $m\theta$. The length condition admits other solutions in principle; the paper picks the simplest and does not claim it is unique.

Many planes, many speeds

A real head has $d = 64$ or $128$ dimensions. RoPE splits the $d$ coordinates into $d/2$ adjacent pairs, $(x_1, x_2), (x_3, x_4), \ldots$, treats each pair as its own plane, and rotates plane $i$ by $m\theta_i$:

A dot product of two $d$-vectors is the sum of the $d/2$ per-plane dot products, and (5) holds in each plane, so the full score is $\mathbf{x}_m^\top\mathbf{W}_q^\top\mathbf{R}^d_{\Theta,n-m}\mathbf{W}_k\mathbf{x}_n$ with $\mathbf{R}^d_{\Theta,n-m} = (\mathbf{R}^d_{\Theta,m})^\top\mathbf{R}^d_{\Theta,n}$ (the paper's Eq. 16). The rates come from the fixed schedule (8), so the projection matrices $\mathbf{W}_q$ and $\mathbf{W}_k$ remain the only learned weights and RoPE adds no parameters. The base 10000 and the geometric spacing are the sinusoidal encoding's; the paper says it follows Vaswani et al. in choosing them.

At $d = 128$ there are 64 planes. Plane 1 turns $\theta_1 = 1$ rad per token, a full turn every $2\pi \approx 6.28$ tokens. Each later plane is slower by a factor $10000^{2/128} \approx 1.155$: plane 13 has a wavelength (tokens per full turn, $2\pi/\theta_i$) of about 35 tokens, plane 29 about 353, plane 47 about 4,700, and plane 64, with $\theta_{64} = 10000^{-126/128} \approx 1.15\times10^{-4}$, about 54,000.

One plane alone cannot locate an offset: at $\theta = 0.1$, offsets 0 and 63 differ by 6.3 rad, almost exactly one full turn, so the plane gives them nearly the same angle. A fast plane separates neighbors (adjacent tokens differ by 1 rad in plane 1) but repeats every 6 tokens; a slow plane does not repeat within any practical context, but adjacent tokens differ by only 0.000115 rad in plane 64. Taken together, the 64 angles pin down the offset at every scale, the way the hour, minute and second hands of a clock fix a time that no single hand could.

Figure 3 shows five of the 64 planes as dials driven by one position. At the default $m = 64$, plane 1 has made about ten turns and plane 64 about a thousandth of one; push the position past 54,000 to watch plane 64 finish its first turn.

Figure 4 tests (3) numerically at $d = 16$, with random token vectors and random projection matrices. A query and a key a fixed offset apart slide along the sequence together. The amber curve adds sinusoidal position vectors before the projection, as in (2); the teal curve applies (7) after it. Change the offset or draw new tokens: the teal spread stays at 0.00 and the amber curve keeps moving with the shift, because the cross terms in (2) depend on $m$ and $n$ separately.

Implementing RoPE without a matrix multiply

$\mathbf{R}^d_{\Theta,m}$ is a $d\times d$ matrix with only $2d$ nonzero entries, so multiplying by it densely would waste almost all of the $d^2$ multiplies. Section 3.4.2 of the paper writes the same product element-wise, with $\otimes$ for element-wise multiplication:

The second vector swaps the two coordinates of each pair and negates the first of them, which is all a 2D rotation needs besides the cosine term. The cosine and sine tables depend only on the position, so they are computed once per sequence length and shared by every layer and head.

A worked example at $d = 8$: four planes with rates $\theta = (1, 0.1, 0.01, 0.001)$. Take the projected query and key

whose plain dot product is 0.35, made of per-plane parts 0.80, −0.75, 0.66 and −0.36. Put the query at $m = 2$: the planes turn by 2, 0.2, 0.02 and 0.002 rad, and (9) gives

Put the key at $n = 5$ and the score is −1.2547. Its four per-plane parts are −0.672, −0.864, 0.642 and −0.360. Plane 4 has turned the pair by only 0.003 rad relative to each other and keeps its unrotated part of −0.36; plane 1 has turned them by 3 rad, and its part has gone from +0.80 to −0.67. Move both tokens to (102, 105) or to (1002, 1005) and the score is −1.2547 again, to every printed digit. Move the key alone to $n = 6$ and the score becomes −1.7763; to $n = 50$, −0.7853.

In code, RoPE is a function applied to the projected queries and keys of every attention layer, just before the scores. The listing uses NumPy-style names and the 0-based index $\theta_i = 10000^{-2i/d}$, $i = 0, \ldots, d/2-1$, which gives the same set of rates as (8).

# RoPE for one attention head. x: (seq_len, d) token states, d even.
def rope(v, base=10000.0):              # v: (seq_len, d), already projected
    seq_len, d = v.shape
    i = arange(d // 2)                  # plane index, 0-based here
    theta = base ** (-2 * i / d)        # (d/2,) fixed rates, theta[0] = 1
    ang = arange(seq_len)[:, None] * theta   # (seq_len, d/2): m * theta_i
    c, s = cos(ang), sin(ang)
    a, b = v[:, 0::2], v[:, 1::2]       # the two coordinates of every plane
    out = empty_like(v)
    out[:, 0::2] = a * c - b * s        # 2D rotation of each pair
    out[:, 1::2] = a * s + b * c
    return out

q = rope(x @ W_q)                       # rotate after projecting
k = rope(x @ W_k)
v = x @ W_v                             # values are never rotated
scores = q @ k.T / sqrt(d)              # entry (m, n) depends on n - m
# no new parameters: gradients reach W_q and W_k through the rotation

The pairing of coordinates differs between code bases. Meta's LLaMA reference code pairs adjacent coordinates, as in (9), by viewing them as complex numbers. Hugging Face's Llama implementation pairs coordinate $j$ with $j + d/2$ (its rotate_half), and its checkpoint conversion script permutes the rows of $\mathbf{W}_q$ and $\mathbf{W}_k$ so that both layouts compute the same scores. Weights trained under one layout and loaded into code using the other without that permutation still run, but they rotate the wrong pairs of coordinates.

Why the attention score decays with distance

Read each plane of the query and key as a complex number, $q_{(i)}$ and $k_{(i)}$. The score (5), summed over planes, is then a sum of $d/2$ rotating terms:

The coefficient $h_i$ holds plane $i$'s content; its real part is that plane's unrotated dot product. (The paper's Eq. 35 conjugates the key instead and gets the exponent $(m-n)\theta_i$; the two sums are complex conjugates with the same real part.)

Set the content aside by taking every $h_i = 1$. What remains is a sum of 64 unit arrows $e^{\mathrm{i}(n-m)\theta_i}$. At offset 0 they all point along the real axis and the sum has length 64. As the offset grows, plane 1's arrow turns 1 rad per token and plane 64's turns 0.000115 rad per token, so the arrows fan out around the circle and partly cancel. Once they point in roughly unrelated directions, the sum behaves like a random walk of 64 unit steps, whose length is of order $\sqrt{64} = 8$. That $\sqrt{d/2}$ level is a heuristic for intuition; the paper's result is the bound below. In Figure 5, drag the offset and compare the bold sum with the dashed circle.

The paper bounds the sum with Abel summation, the discrete version of integration by parts. Let $S_j$ be the partial sum of the first $j$ unit arrows. Each term of (10) is $h_i$ times a difference of consecutive partial sums, and regrouping the sum around the partial sums (with $h_{d/2+1} = 0$) moves the differences onto the $h_i$. The triangle inequality then gives (the paper's Eq. 37, with 1-based indices):

The first factor depends only on the content of the query and key. The second depends only on the offset and the rates. The paper plots its average, $\tfrac{1}{d/2}\sum_j |S_j|$, against the offset (its Figure 2), observes that it decays, and ties this to the intuition that “a pair of tokens with a long relative distance should have less connection”. Figure 6 plots the same factor, normalized to 1 at offset 0 (before normalizing it starts at $(d/2+1)/2 \approx d/4$, since $|S_j| = j$ there).

At base 10000 the factor halves within the first 16 tokens and then oscillates on a slowly falling trend. A larger base slows every plane and moves the fall to longer offsets. Two limits on what (11) says: it bounds the size of the score and does not predict it, so a query and key with matching content can still score high at long range; and the geometric factor does not fall monotonically, because at some offsets several planes come back into partial alignment and the factor rises for a while.

RoPE inside linear attention

Softmax attention (1) forms all $N^2$ query-key scores for a sequence of $N$ tokens. Linear attention (Katharopoulos et al., 2020) replaces $\exp(\mathbf{q}^\top\mathbf{k}/\sqrt{d})$ by a product of feature maps $\phi(\mathbf{q})^\top\phi(\mathbf{k})$, with the positive map $\phi(x) = \mathrm{elu}(x) + 1$. Attention becomes (the paper's Eq. 18)

and matrix products can be regrouped. Written as a row vector, the numerator is $\phi(\mathbf{q}_m)^\top \mathbf{S}$ with

and the denominator is $\phi(\mathbf{q}_m)^\top\mathbf{z}$. $\mathbf{S}$ and $\mathbf{z}$ do not depend on $m$: one pass over the keys builds them, and each query then costs one vector-matrix product, so the total grows as $N$ instead of $N^2$. At $N = 4096$ and $d = 64$, softmax attention forms about 16.8 million scores per head, while linear attention adds 4,096 outer products into one $64\times 64$ matrix. For causal attention each query may only see keys up to its own position, so $\mathbf{S}$ becomes a running sum over $n \le m$: still one pass, but each query reads its own prefix sum rather than one shared matrix.

An additive bias does not survive this regrouping. Inside softmax attention, a bias $b_{m,n}$ turns $\exp(\mathbf{q}_m^\top\mathbf{k}_n)$ into $\exp(\mathbf{q}_m^\top\mathbf{k}_n)\,e^{b_{m,n}}$, a factor that depends on the query position and the key position together. The linear-attention analogue would multiply each term of (12) by a pair-dependent number, which cannot be moved into a sum shared by all queries, so every pair has to be visited again. This argument is ours; RoFormer only contrasts its multiplicative scheme with the additive ones.

A rotation acts on each vector separately. The paper's Eq. 19 applies it to the outputs of the feature maps and leaves the denominator unrotated:

In the numerator $(\mathbf{R}^d_{\Theta,m})^\top\mathbf{R}^d_{\Theta,n} = \mathbf{R}^d_{\Theta,n-m}$ again, so each pair's weight is $\phi(\mathbf{q}_m)^\top\mathbf{R}^d_{\Theta,n-m}\phi(\mathbf{k}_n)$, relative for any choice of $\phi$. The accumulator survives because the query's rotation does not depend on $n$ and moves out of the sum. The paper prints only (14); the factored form is our derivation:

The bracket is built in one pass with each key rotated by its own position, and each query is rotated by its own position before reading it, so the cost stays linear in $N$. Rotation can make some weights $\phi(\mathbf{q}_m)^\top\mathbf{R}^d_{\Theta,n-m}\phi(\mathbf{k}_n)$ negative even though $\phi$ is positive, so the paper keeps the denominator unrotated “to avoid the risk of dividing zero” and says the resulting weights are “not strictly probabilistic normalized”. Figure 7 contrasts the two data flows.

What the paper measured

Translation. On WMT 2014 English to German (about 4.5 million sentence pairs), a Transformer-base trained in fairseq scores 27.3 BLEU and the same model with RoPE scores 27.5 (Table 1). The paper reports no variance across runs, so the 0.2-point difference has no error bar.

Pre-training. The authors trained bert-base-uncased and the same model with RoPE on BookCorpus and Wikipedia (batch 64, sequences of 512, 100k steps, AdamW at 1e-5), and the RoPE model's masked-language-model loss falls faster (Figure 3, left). The paper describes this as replacing BERT's “sinusoidal” position encoding, but BERT's position embeddings are learned, as the paper's own Section 2.2 says.

GLUE. Fine-tuning that model gives Table 2:

RoFormer is higher on MRPC, STS-B and QQP and lower on SST-2, QNLI and MNLI. The two rows also come from different evaluation sets. The BERT row is, digit for digit, BERT-base's GLUE test-server result from Devlin et al.'s Table 1, while Section 4.3.2 says RoFormer's numbers are “the best-averaged results on the validation set”. The 15-point QQP gap therefore cannot be credited to RoPE, and the table supports no conclusion stronger than “roughly comparable”.

Linear attention. On enwik8 (character-level Wikipedia), a 12-layer Performer with 768 dimensions and 12 heads, at sequence length 1024, reaches a lower training loss sooner with RoPE than without (Figure 3, right). This is a loss curve with and without RoPE inside one linear-attention model; there is no bits-per-character table and no comparison with BERT.

Long Chinese documents. The authors pre-trained a Chinese RoFormer on about 34GB of text in six stages with maximum lengths between 128 and 1,536, then evaluated on CAIL2019-SCM (Xiao et al., 2019): 8,964 triplets of court-case descriptions, mostly longer than 512 characters, where the task is to decide whether case B or case C is closer to case A. Test accuracy (Table 5) is 67.77% for BERT-512, 68.10% for WoBERT-512, 68.29% for RoFormer-512 and 69.79% for RoFormer-1024. At the same 512-token input RoFormer is 0.19 points above WoBERT; the larger gain comes with doubling the input to 1,024 tokens, which the absolute-position baselines were not run at. The text calls this a 1.5% improvement; Table 5 gives 1.69 points on test and 2.00 on validation.

The paper's limitations section concedes that it lacks “thorough explanations” for why RoFormer converges faster, and that for the long-text results “we have not come up with a faithful explanation”.

RoPE spread through its engineering properties rather than through these results. It has no parameters, so there is no position table to outgrow and nothing position-specific to fine-tune. It is applied to the query and key tensors before attention, so it composes with linear attention as above and runs unchanged inside exact attention kernels such as FlashAttention. Its rates and positions are explicit numbers that can be changed after training, which the next section relies on. GPT-NeoX, PaLM, LLaMA and Mistral use RoPE; GPT-3 and OPT used learned absolute positions and BLOOM used ALiBi, a fixed linear distance penalty added to each score.

Running RoPE past its training length

Call the trained context $L$ and the target $L'$, and the extension factor $s = L'/L$. Llama 2 was trained at $L = 4096$; running it at 32,768 tokens means $s = 8$. Unchanged RoPE fails there in a specific way, which the YaRN paper describes in terms of wavelengths. With $d = 128$, planes 47 through 64 have wavelengths longer than 4,096 tokens, so during training those 18 planes never completed a turn: plane $i$ only produced angles between 0 and $4096\,\theta_i$. At position 30,000 the same planes reach angles the model was never trained on. The fast planes complete many turns within 4,096 tokens, so every angle they reach at a longer context already occurred in training. The three patches below all slow the slow planes and differ in what they do to the fast ones.

Position Interpolation (Chen et al., 2023) divides every position by $s$, which divides every rate by $s$. Every angle at the new length then falls inside the trained range. Dividing every rate also lowers resolution: at $s = 8$, plane 1 turns 0.125 rad per token instead of the 1 rad it turned in training, so adjacent tokens are eight times closer in angle in that plane. PI extended LLaMA models to 32,768 tokens with a fine-tune of at most 1,000 steps.

NTK-aware scaling (bloc97, 2023, written up in the YaRN paper's Appendix A.2) raises the base instead:

which slows plane $i$ by the factor $s^{2(i-1)/(d-2)}$. Plane 1 is unchanged, plane 64 is slowed by exactly $s$, and the planes in between are slowed by less than $s$. At $s = 4$, $d = 128$ the base becomes $10000 \times 4.09$. The fast planes keep their resolution, but planes 47 to 63 are slowed by less than $s$ and still reach angles beyond their trained range.

YaRN (Peng et al., 2023) treats each plane according to how many turns it makes in the trained context, $r_i = L/\lambda_i$ with wavelength $\lambda_i = 2\pi/\theta_i$. A ramp $\gamma(r)$ is 0 for $r < \alpha$, 1 for $r > \beta$ and linear in between, and each rate becomes

so planes that turn more than 32 times are left alone, planes that turn less than once are divided by $s$ as in PI, and planes in between are blended. The YaRN paper calls this rescaling “NTK-by-parts”, and YaRN is NTK-by-parts plus an attention temperature: queries and keys are both scaled by $\sqrt{1/t} = 0.1\ln s + 1$, which is 1.208 at $s = 8$. YaRN does not change the base. It extended Llama 2 to 128k tokens with $s = 32$.

Figure 8 plots all 64 wavelengths before and after each patch. At $s = 8$: unchanged RoPE has 18 planes past their trained angles and plane 1 at 1 rad per token; NTK-aware has 17 such planes and plane 1 at 1 rad; PI has none, with plane 1 at 0.125 rad; YaRN has none, with plane 1 at 1 rad.

None of the three changes the RoFormer construction. Each one changes the rates $\theta_i$ or the positions fed into (7), and (5) holds for any rates, so scores still depend only on the offset (scaled by $1/s$ under PI). Each patch is usually followed by a short fine-tune at the new length: at most 1,000 steps for PI's LLaMA models and 400 steps for YaRN's $s = 16$ Llama 2 model.