In deep learning, the positional encoding or position embedding is a component of sequence models, especially Transformers. Input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and the position of each token is encoded as a vector. Positional encoding is used because the standard attention mechanism used in Transformers by default treats all tokens symmetrically over all positions (i.e. permutation-symmetric), thus effectively treating the input as a bag of tokens, rather than a sequence of tokens.
Absolute vs relative positional encoding:
Learned vs designed positional encoding:
For example, the original Transformer used designed absolute positional encoding, and the GPT transformer used learned absolute positional encoding. BERT used two levels of positional embeddings, one for the per-token position, and one for per-segment position. Both are absolute and learned. It was reported that learned absolute positional encoding generally performs better than relative positional encoding when there are special tokens that always occur in a specific location, such as the [CLS] token of BERT-like architectures. However, with learned absolute positional encoding, a model cannot process contexts longer than what it is trained for.
Other possibly desirable properties of a positional encoding scheme include:

For the original transformer (Attention Is All You Need), published in 2017, each token is converted to a token embedding vector , plus a positional encoding vector .
depends only on the identity of the token, not on . That is, the token "the" would have the same , no matter if it appears in the first or the last position. depends only on the position of the token, not on the identity of the token. That is, the -th token always has the same , no matter if it is "the" or "a".
Both the encoder and the decoder take as input. This is shown in the picture.
Both and have the same number of dimensions, since they must be added. Both of them have even dimensions, since that is what the positional encoding vector requires.
The original paper reported two methods of obtaining . One was the learned absolute positional encoding previously used in convolutional seq2seq. The other was the designed, sinusoidal absolute positional encoding.
Let the dimension be , then the position encoding vector is defined as

where .
Here, is the base frequency, a free parameter that should be significantly larger than the biggest that would be input into the positional encoding function (normally, this is the context length). The original paper uses .
More succinctly,In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.
The positional encoding has been interpreted as implementing a base- positional number system for representing real numbers:
| Position | Frequency | Factor | Tokens per cycle ("wavelength") | Tokens per cycle at |
|---|---|---|---|---|
| Lowest | 1 | 6.3 | ||
| Next lowest | 7.5 | |||
| Highest | 52261 |
The effect of base- positional number system means the attention mechanism is to apportion its attention by coordinate. For low frequency (long wavelength) coordinates, the encoding implicitly states that these coordinates encode information that varies slowly over a sequence, and vice versa for high frequency (low wavelength) coordinates. Concretely, the first 2 coordinates are treated as encoding features that vary quickly every token, while the last 2 coordinates are treated as encoding features that vary slowly over 10000s of tokens.
This positional encoding method was designed based on previous work on convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."
A relative position shift is just a linear transformation away:where is the distance one wishes to shift. More generally, any convolution of relative positions is also just a linear transformation away:for any constants . This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors.
In the original Transformer, positions are encoded by a vector that is added to the token embedding vector , and the Transformer only sees their sum . Several subsequent works produced variations of it. Compared to the original positional encoding scheme, they are relative, additive, and partially decouple the position encoding from the token embedding, by injecting positional encoding directly into each attention mechanism block, rather than adding only to the input.
The attention mechanism block in the original Transformer works as follows. It takes two input sequences , and , then computes three sequencesthen computes attention logits by taking a dot-product, multiplying by a positive scalar , then adding a mask:then computes attention weights by taking a softmax:then computes output vectors by taking a weighted sum of the value vectors:This is more succinctly written as The scalar is usually selected to be where is the dimension of the key vectors .
Relative Position Representation (RPR) was proposed in 2018, and two of the coauthors were coauthors of the original Transformer paper. In RPR, three choices were relative, added to the key and value vectors, and learned.
Concretely, the attention logits are calculated as and the output vectors are calculated asThe key PE vectors and value PE vectors are learned. This is a learned dictionary of embedding vectors for relative positions, the same way as the embedding layer of a Transformer, which is a learned dictionary of embedding vectors for tokens.
The authors additionally clamped the vectors, i.e. for large enough , they designed the identification , and similarly for the value PE vectors, and for small enough . This was designed so as to enforce an inductive bias that "relative position information is not useful beyond a certain distance". That is, the positional encoding scheme does not distinguish between "100 tokens apart" and "101 tokens apart". The authors also claimed that this allowed the model to generalize to sequence lengths longer than those seen during training.
The deBERTa transformer was a variant of BERT. It adds relative positional encoding to the query and key vectors.It does not include the 4th term , since this term is a bias term depending purely on position, not token content.
It does not use the masking matrix since it is an all-to-all attention, like the original BERT.
XLNet transformer adds relative positional encoding to the query and key vectors, and then identifies some of them. This was based on the previous Transformer-XL, which uses the same formula, except absolute positional encoding instead of relative positional encoding.
Begin with the expressionwhere play the same roles as , except they map the position encoding vectors, rather than the token encoding vectors. Then expand the brackets to 4 terms, and identify some of the vectors, to obtain
ALiBi (Attention with Linear Biases) adds a bias term to encode the relative position.Here, is a scalar, and is an entry in the linear bias matrix defined byin other words, . The idea being that the linear bias matrix is a softened mask. Just as represent full attention paid, and represents no attention paid, the linear bias matrix linearly decreases the attention logit. Note that in the causally masked Transformer, any attention paid to the future is blanked out by the causal mask matrixso the upper triangular part of does not matter.
The original paper designed . Each attention head gets a different one. The first attention head gets , the second attention head gets , and so on.
ALiBi allows pretraining on short context windows, then fine-tuning on longer context windows. Since it is directly plugged into the attention mechanism, it can be combined with any positional encoder that is plugged into the "bottom" of the entire network (which is where the sinusoidal encoder on the original transformer, as well as RoPE and many others, are located).
More generally, can be a Toeplitz matrix.with learned entries. The scalar can also be learned.
The T5 transformer series used the more general version of ALiBi. Like the original RPR, it also groups the entries into "ranges". This enforces an inductive bias that, as positions become further apart, the resolution decreases. It is important to distinguish between 1 and 2, but less important to distinguish 10 and 12, etc. As the original authors state, the ranges "increase in size logarithmically up to an offset of 128 beyond which we assign all relative positions to the same embedding".
In detail, the T5 encoder encodes as follows: , then the entries are distinct, then , and so on, until unto infinity. Same on the other direction with etc.
RoPE (Rotary Positional Encoding) is another scheme for relative positional encoding. It works by rotating the query and key vectors input into an attention mechanism block. It was proposed in 2021 in a Chinese blog, and then verified experimentally by Eleuther AI. Eleuther AI used it in their GPT-NeoX in early 2022, and it subsequently spread in popularity.
Consider a feature vector with even dimensions: . Like in the original sinusoidal positional encoding scheme, it can be written as a complex vector . Then, to encode it as positioned at , the -th feature vector is represented by rotating its entries:where the frequencies are usually designed by hand. A standard design is where is the base frequency hyperparameter, as in the original Transformer.
In the original RoPE paper and Llama, . Larger values of are often used for models with longer contexts. For example, during the training of Llama 3, the context length was 8192, but its , to "better support longer contexts".
More succinctly, we define to be the diagonal matrix of , so that rotating a vector is the same as multiplying by the matrix on the right, as in . In real space, it is converted to a block-diagonal matrix, each block being 2×2.
In an attention mechanism, RoPE is implemented asThis exploits properties of the rotation matrix: and .
Like the original positional encoding, it behaves as a base- positional number system.
Several variants begin by an observation about RoPE. In the equationthe RoPE part is in the middle, and by the translational invariance of RoPE, is equal to . The idea is to modify the term.
These variants have seen use in context extension.
Extrapolatable Position Embedding (xPos) adds decay terms to the rotation of RoPE, designed to damp out the high-frequency oscillations at long distances.This, as before, is represented as a diagonal matrix in complex space, which is equivalent to a 2×2 block-diagonal matrix in real space.The original paper suggested . xPos is also limited to causal attention masking, since otherwise the attention logits will diverge.
RoPE-2D applies RoPE to tokens that are arranged on a plane. They are used in vision transformers. A vector representing a token at location is split into two halves, the first half rotated by , and the second half rotated by . Here, and do not have to be the same sequence of angular frequencies.
Similarly, Transformers that process videos may use RoPE-3D. It can also be used for other purposes. For example, the vision-language model Qwen-VL uses a version of RoPE-3D that they call "M-RoPE" (multimodal RoPE). Each token is given a triple index . A video is divided into individual frames, then patches. The first index increments every image, frame, or textual token. For a patch, count the horizontal and vertical position. For a textual token, .
RoPE with Relative distances (RoPER), or value-output RoPE (VO-RoPE), applies RoPE to value vectors.

In NoPE (No Positional Encoding), there is no positional encoding. The idea is that a decoder's causal masking lets each token to see only tokens , which is an implicit kind of positional encoding. Relatedly, pure convolutional networks are capable of implicit absolute positional encoding, by exploiting the zero-padding on the edge.
NoPE can represent both absolute and relative positional encodings. Empirically, when trained with gradient descent, a NoPE transformer and a transformer using T5-style positional encoding (ALiBi, learned) produce similar distributions of attention weights for the same input sequences, measured using Jensen–Shannon divergence. It was also shown theoretically and empirically that a trained NoPE transformer represents the first few tokens distinctly in the representational space compared to other positions, which it is capable of distinguishing via causal masking.
NoPE is rarely used for all attention modules of a Transformer, since it would degrade performance significantly. However, NoPE may be used for a fraction of attention modules. This was found in practice to be a worthwhile trade off of performance for computing cost. For example, Llama 4 alternates layers of NoPE and RoPE, with RoPE in 3 layers and NoPE in the 4th. They call it iRoPE ("interleaved RoPE").
Since the per-token cost of inference scales with context length, training cost is decreased by using shorter contexts during training. However, this would make the resulting model bad at using longer contexts. Consequently, a common way to train a model is to "train short, test long", i.e. training a model on short contexts, and use it on long contexts.
Context extension faces two types of distributional shift:
Context extension was a concern since the original Transformer paper. In it, the authors reported trying both learned positional encoding and designed positional encoding. They found both to result in similar performance, and decided to use the designed positional encoding, since it "may allow the model to extrapolate to sequence lengths longer than the ones encountered during training".
A slightly more sophisticated method is to train according to a curriculum: Begin by training on a short context length, then train on a longer context length. This can be repeated. For example, the model can be pretrained on context lengths up to 8K tokens, then up to 64K, then up to 512K.
Context lengths can even be mixed during training. It was found that given a fixed training budget, training on a mixture of long contexts and short contexts resulted in higher performance than purely training on long contexts.
In general, context extension using RoPE consists of training a model on one context length with one set of RoPE angles , then its RoPE angles are modified to , and it is trained further on context length for some .
Some architectures natively support context extension. For example, convolutional seq2seq, and sliding window attention, are both insensitive to context length.
YaRN (Yet another RoPE extensioN) is a common way to extend context, used across many models, including DeepSeek, Qwen, Kimi, and GPT-OSS. It was built in steps.
In Positional Interpolation (PI), frequencies are stretched equally: . This was found to degrade performance beyond , as the model loses the high-frequency resolution that tells adjacent tokens apart.
In Adjusted Base Frequency (ABF) (other names include "NTK-aware", "NTK-RoPE", "RoPE-ABF"), the base is increased: , where is designed so thatEquivalently, . The effect is that the highest frequency is not stretched, the lowest frequency is stretched by , and those in-between are interpolated.
In "NTK-by-parts", high frequencies are not stretched, low frequencies are stretched by , and those in-between are interpolated:where
YaRN uses NTK-by-parts with frequency scales , meaning that wavelengths shorter than are not stretched, wavelengths longer than are fully stretched by , and those in-between are stretched by an in-between value.
YaRN also scales up the logits in the attention mechanism. A longer context results in more entries to spread the attention weights, thus tends to produce flatter attention weights than shorter contexts. Scaling up the logits un-flattens them. The authors experimented with context-extending models from the Llama and Llama 2 series, and found empirically that worked well.
Because extending the context length involves a degradation of performance, during deployment, a model is used with its original context length , and when the context exceeds , switch to the extended context length . This was found to cause a sudden drop of performance at the switching point. Consequently, dynamic scaling ("dynamic NTK") was used to allow graceful degradation.
The algorithm:
Dynamic scaling can be used even with no extra training on extended context . It was first proposed in a post in the subreddit r/LocalLLaMA.