Transformer (deep learning)

In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which input data such as text, images, or audio is converted to a sequence of numerical representations called tokens, and each token is converted into a vector through an embedding layer. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Because self-attention alone is permutation-invariant, transformers inject positional information, typically through positional encodings or learned positional embeddings, so token order can affect the output.

Because transformers do not process tokens one at a time, their computations can be parallelized across sequence positions during training more readily than those of recurrent neural networks like long short-term memory (LSTM). Later variations have been widely adopted for training large language models (LLMs) on large (language) datasets.^{[citation needed]}

The original version of the transformer architecture was proposed in the 2017 paper "Attention Is All You Need" by researchers at Google. The predecessors of transformers were developed as an improvement over previous architectures for machine translation, but have found many applications since. They are used in large-scale natural language processing, computer vision (vision transformers), reinforcement learning, audio, multimodal learning, robotics, and playing chess. It has also led to the development of pre-trained systems, such as generative pre-trained transformers (GPTs).

History

Predecessors

Before transformers, recurrent neural networks (RNNs) were widely used for sequence modelling and generation. A well-cited early example was the Elman network (1990). In theory, the information from one token can propagate arbitrarily far down the sequence, but in practice the vanishing-gradient problem leaves the model's state at the end of a long sentence without precise, extractable information about preceding tokens.

A key breakthrough was LSTM (originally described in a 1995 technical report and formally published in 1997), an RNN that introduced gating mechanisms to mitigate the vanishing gradient problem, allowing efficient learning of long-sequence modelling. One key architectural element was the use of multiplicative gating units, in which the outputs of some neurons modulate the outputs of others. These multiplicative units are conceptually distinct from the additive attention mechanism later introduced for sequence-to-sequence models. Neural networks using multiplicative units were later called sigma-pi networks or higher-order networks. LSTM became the standard architecture for long sequence modelling until the 2017 publication of transformers. However, LSTM still used sequential processing, like most other RNNs. Specifically, RNNs operate one token at a time from first to last; they cannot operate in parallel over all tokens in a sequence.

Transformers allow parallel processing of tokens during training, but standard self-attention has computational and memory costs that grow quadratically with sequence length.

Attention with seq2seq

The idea of encoder–decoder sequence transduction had been developed in the early 2010s; commonly cited as the originators that produced seq2seq are two concurrently published papers from 2014.^{[original research?]}

A 380M-parameter model for machine translation uses two long short-term memories (LSTM). Its architecture consists of two parts. The encoder is an LSTM that takes in a sequence of tokens and turns it into a vector. The decoder is another LSTM that converts the vector into a sequence of tokens. Similarly, another 130M-parameter model used gated recurrent units (GRU) instead of LSTM. Later research showed that GRUs are neither better nor worse than LSTMs for seq2seq.

These early seq2seq models had no attention mechanism, and the state vector is accessible only after the last word of the source text was processed. Although in theory such a vector retains the information about the whole original sentence, in practice the information is poorly preserved. This is because the input is processed sequentially by one recurrent network into a fixed-size output vector, which is then processed by another recurrent network into an output. If the input is long, then the output vector would not be able to contain all relevant information, degrading the output. As evidence, reversing the input sentence improved seq2seq translation.

The RNN search model introduced an attention mechanism to seq2seq for machine translation to solve the bottleneck problem (of the fixed-size output vector), allowing the model to process long-distance dependencies more easily. The name is because it "emulates searching through a source sentence during decoding a translation".

The relative performances were compared between global (that of RNN search) and local (sliding window) attention model architectures for machine translation, finding that mixed attention had higher quality than global attention, while local attention reduced translation time.

In 2016, Google Translate was revamped to Google Neural Machine Translation, which replaced the previous model based on statistical machine translation. The GNMT system was a sequence-to-sequence model with eight LSTM layers in its encoder and eight in its decoder. The first encoder layer was bidirectional, the remaining seven encoder layers were unidirectional, and all eight decoder layers were unidirectional. It took nine months to develop, and it outperformed the statistical approach, which took ten years to develop.

Parallelizing attention

Seq2seq models with attention (including self-attention) still suffered from the same issue with recurrent networks, which is that they are hard to parallelize, which prevented them from being accelerated on GPUs. In 2016, decomposable attention applied a self-attention mechanism to feedforward networks, which are easy to parallelize, and achieved state-of-the-art (SOTA) results in textual entailment with an order of magnitude fewer parameters than LSTMs. One of its authors, Jakob Uszkoreit, suspected that attention without recurrence would be sufficient for language translation, thus the title "attention is All you need". That hypothesis was against conventional wisdom at the time, and even his father Hans Uszkoreit, a well-known computational linguist, was skeptical. In the same year, self-attention (called intra-attention or intra-sentence attention) was proposed for LSTMs.

The original encoder–decoder Transformer was introduced in the paper "Attention Is All You Need," submitted to arXiv on 2017-06-12. At the time, the focus of the research was on improving seq2seq for machine translation, by removing its recurrence to process all tokens in parallel, but preserving its dot-product attention mechanism to keep its text processing performance. This led to the introduction of a multi-head attention model that was easier to parallelize due to the use of independent heads and the lack of recurrence. Its parallelizability was an important factor to its widespread use in large neural networks.

AI boom era

Soon after developing the Transformer, researchers at Google used a decoder-only variation to generate fictitious Wikipedia articles. Transformer architecture is now used alongside many generative models that contribute to the ongoing AI boom.

The "reference implementation" of the original Transformer was written in a TensorFlow library. In language modelling, ELMo (2018) was a bi-directional LSTM that produces contextualized word embeddings, improving upon the line of research from bag of words and word2vec. It was followed by BERT (2018), an encoder-only transformer model. In October 2019, Google started using BERT to process search queries. In 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model by a transformer-encoder–RNN-decoder model.^{[non-primary source needed]}

Starting in 2018, the OpenAI GPT series of decoder-only transformers became state of the art in natural language generation. At the end of 2022, ChatGPT, a chatbot based on a fine-tuned variant of GPT-3.5, became unexpectedly popular, triggering a boom around large language models.

Transformers have been applied in modalities beyond text. Four days after the publication of "Attention is All You Need", a multimodal transformer architecture, MultiModel, was published by most authors of that paper. The vision transformer adapts the architecture to images by processing image patches as a sequence of tokens. Transformer models are also used in systems for image and video generation, although these systems differ in how they represent and process their inputs and outputs.

Training

Methods for stabilizing training

The original Transformer paper used a learning-rate warmup schedule: the learning rate increased linearly for the first 4,000 training steps and then decayed according to an inverse square-root schedule.

A study of the Transformer found that applying layer normalization before the attention and feed-forward sublayers can improve training stability and reduce dependence on learning-rate warmup in some settings. This arrangement is called pre-LN; the original Transformer used post-LN.

Regularization

The original transformer used dropout during training to reduce overfitting. Dropout was applied to the output of each sublayer before its residual connection and normalization, as well as to the sums of token embeddings and positional encodings. The original paper also used label smoothing, which replaces a certain target distribution with one that assigns a small amount of probability to alternative tokens. These techniques can improve generalization by going against overconfidence in the training targets.

Distributed Training

Large transformer models are typically trained across multiple GPUs or other accelerators because their parameters, activations, and training data may not fit on a single device. Data parallelism copies the model across devices and splits different batches of training examples among them; gradients are then synchronized before parameters are updated. Model parallelism divides a model itself across devices. Tensor parallelism splits operations within individual layers, while pipeline parallelism assigns consecutive groups of layers to different devices.

Pretrain-finetune

Many transformer-based language models are pretrained on large corpora using self-supervised objectives, and some are later fine-tuned for specific tasks. The dataset and objective vary by model. Common pretraining objectives include:

next-token prediction in autoregressive language models
masked-token prediction
span corruption or denoising, used by T5.

Downstream fine-tuning tasks can include reading comprehension, sentiment classification, paraphrasing, and summarization.

The T5 transformer report documents a large number of natural language pretraining tasks. Some examples are:

restoring or repairing incomplete or corrupted text. For example, the input, "Thank you ~~ me to your party ~~ week", might generate the output, "Thank you for inviting me to your party last week".
translation between natural languages (machine translation)
judging the pragmatic acceptability of natural language. For example, the following sentence might be judged "not acceptable", because even though it is syntactically well-formed, it is improbable in ordinary human usage: The course is jumping well.

While each of these tasks is trivial or obvious for human native speakers of the language (or languages), they have typically proved challenging for previous generations of machine learning architecture.

Knowledge distillation

Knowledge distillation is a method for training a smaller model, called a "student", to reproduce the output behavior of a larger "teacher" model. In transformer models, the student can be trained using the teacher's output probability distribution, intermediate representations, or attention patterns in addition to the original training labels. Distillation can lower computational and memory requirements of deployment while keeping most of the teacher model's performance on a specified task.^{[citation needed]}

Tasks

In general, there are three classes of language modelling tasks: "masked", "autoregressive", and "prefixLM". These classes are independent of a specific modeling architecture such as transformer, but they are often discussed in the context of transformer.

In a masked task, one or more of the tokens is masked out, and the model would produce a probability distribution predicting what the masked-out tokens are based on the context. The loss function for the task is typically sum of log-perplexities for the masked-out tokens: $Loss = - \sum_{t \in masked tokens} \ln (probability of t conditional on its context)$ and the model is trained to minimize this loss function. The BERT series of models are trained for masked token prediction and another task. ("Masked" as in "masked language modelling" is not "masked" as in "masked attention".)

In an autoregressive task, the model predicts each token from the preceding context. During training, a causal attention mask prevents each position from attending to later tokens, allowing the sequence to be processed in parallel while the loss is computed for target tokens. During inference, tokens are generated one or more at a time and appended to the context. The GPT series of models are trained with autoregressive objectives.

In a prefix-language-modeling task, the model receives a prefix as context and predicts a continuation. Depending on the architecture and mask, prefix tokens may attend to one another bidirectionally while each continuation token can attend only to the prefix and earlier continuation tokens. T5 is an encoder–decoder model trained primarily with span-corruption objectives, rather than this prefix-LM objective.

Architecture

Transformer architectures vary, but many include some of the following components:

A tokenizer, which maps input data to tokens or other discrete units.
An embedding or input-projection layer, which maps those units to vectors and may add positional information.
One or more attention and feed-forward layers. Encoder-only, decoder-only, and encoder–decoder models use different arrangements of these layers.
An output layer appropriate to the task, such as a vocabulary projection for language modeling.

The following description focuses on the encoder–decoder Transformer introduced in the original paper; other variants are described in the following section.

By convention, we write all vectors as row vectors. For example, pushing a vector through a linear layer means multiplying it by a weight matrix on the right, as $x W$ .

Tokenization

Tokenization maps input text to a sequence of token identifiers drawn from a vocabulary. Depending on the tokenizer, preprocessing may normalize text and divide it into words, subwords, or bytes. Subword methods such as byte pair encoding (BPE) and the unigram language model can represent many rare or previously unseen words as sequences of known units, rather than requiring a single [UNK] token. A tokenizer's vocabulary is learned or otherwise specified before it is used.

Embedding

Each integer token identifier is converted into an embedding vector via a lookup table. Equivalently stated, it multiplies a one-hot representation of the token identifier by an embedding matrix $M$ . For example, if the input token's identifier is $3$ , then the one-hot representation is $[0, 0, 0, 1, 0, 0, \dots]$ , and its embedding vector is $E m b e d (3) = [0, 0, 0, 1, 0, 0, \dots] M$ The token embedding vectors are added to their respective positional encoding vectors (see below), producing the sequence of input vectors.

The dimension of an embedding vector is called hidden size or embedding size and written as $d_{emb}$ . This size is written as $d_{model}$ in the original transformer paper.

Un-embedding

An un-embedding layer is almost the reverse of an embedding layer. Whereas an embedding layer converts a token identifier into a vector, an un-embedding layer converts a vector into a probability distribution over tokens.

The un-embedding layer is a linear-softmax layer: $U n E m b e d (x) = s o f t m a x (x W + b)$ The matrix has shape $(d_{emb}, | V |)$ . Some architectures use the transpose of the embedding matrix $M$ as the un-embedding matrix $W$ in order to avoid needing double the amount of embedding-related parameters and to avoid divergence during training. This practice is called weight tying.

Positional encoding

A positional encoding or embedding supplies information about token position or order. The original Transformer added fixed sinusoidal position encodings to token embeddings and reported similar results with learned positional embeddings; later models use other positional schemes. This induces a bias towards the order of the input sequence, so that, for example, the input sequence "man bites dog" is processed differently from "dog bites man".

The positional encoding is defined as a function of type $f : ℝ \to ℝ^{d}$ , where $d$ is a positive even integer. The full positional encoding defined in the original paper is: $(f (t)_{2 k}, f (t)_{2 k + 1}) = (\sin (θ), \cos (θ)) \forall k \in {0, 1, \dots, d / 2 - 1}$ where $θ = \frac{t}{r^{k}}, r = N^{2 / d}$ .

Here, $t$ denotes a sequence position and $k$ indexes pairs of embedding dimensions. $N$ controls the range of sinusoidal wavelengths; the original paper used $N = 10000$ .

The function is in a simpler form when written as a complex function of type $f : ℝ \to ℂ^{d / 2}$ $f (t) = {(e^{i t / r^{k}})}_{k = 0, 1, \dots, \frac{d}{2} - 1}$ where $r = N^{2 / d}$ .

The main reason for using this positional encoding function is that using it, shifts are linear transformations: $f (t + Δ t) = d i a g (f (Δ t)) f (t)$ where $Δ t \in ℝ$ is the distance one wishes to shift. This allows the transformer to take any encoded position, and find the encoding of the position n-steps-ahead or n-steps-behind, by a matrix multiplication.

By taking a linear sum, any convolution can also be implemented as linear transformations: $\sum_{j} c_{j} f (t + Δ t_{j}) = (\sum_{j} c_{j} d i a g (f (Δ t_{j}))) f (t)$ for any constants $c_{j}$ . This allows the transformer to take any encoded position and find a linear sum of the encoded locations of its neighbors. This sum of encoded positions, when fed into the attention mechanism, would create attention weights on its neighbors, much like what happens in a convolutional neural network language model. In the author's words, "we hypothesized it would allow the model to easily learn to attend by relative position."

In typical implementations, all operations are done over the real numbers, not the complex numbers, but since complex multiplication can be implemented as real 2-by-2 matrix multiplication, this is a mere notational difference.

Encoder–decoder (overview)

Like earlier seq2seq models, the original transformer model used an encoder–decoder architecture. The encoder consists of encoding layers that process all the input tokens together one layer after another, while the decoder consists of decoding layers that iteratively process the encoder's output and the decoder's output tokens so far.

The purpose of each encoder layer is to create contextualized representations of the tokens, where each representation corresponds to a token that "mixes" information from other input tokens via self-attention mechanism. Each decoder layer contains two attention sublayers: (1) cross-attention for incorporating the output of encoder (contextualized input token representations), and (2) self-attention for "mixing" information among the input tokens to the decoder (i.e. the tokens generated so far during inference time).^{[citation needed]}

Both the encoder and decoder layers have a feed-forward neural network for additional processing of their outputs and contain residual connections and layer normalization steps. These feed-forward layers contain most of the parameters in a transformer model.^{[citation needed]}

Feedforward network

The feedforward network (FFN) modules in a transformer are 2-layered multilayer perceptrons: $F F N (x) = ϕ (x W^{(1)} + b^{(1)}) W^{(2)} + b^{(2)}$ where $W^{(1)}$ and $W^{(2)}$ are weight matrices and $b^{(1)}$ and $b^{(2)}$ are bias vectors, and $ϕ$ is its activation function. The original transformer used ReLU activation.

The number of neurons in the middle layer is called intermediate size (GPT), filter size (BERT), or feedforward size (BERT). It is typically larger than the embedding size. For example, in both GPT-2 series and BERT series, the intermediate size of a model is 4 times its embedding size: $d_{ffn} = 4 d_{emb}$ .

Scaled dot-product attention

Attention head

The attention mechanism used in the transformer architecture are scaled dot-product attention units. For each unit, the transformer model learns three weight matrices: the query weights $W^{Q}$ , the key weights $W^{K}$ , and the value weights $W^{V}$ .

The module takes three sequences, a query sequence, a key sequence, and a value sequence. The query sequence is a sequence of length $ℓ_{seq, query}$ , and each entry is a vector of dimension $d_{emb, query}$ . Similarly for the key and value sequences.

For each vector $x_{i, query}$ in the query sequence, it is multiplied by a matrix $W^{Q}$ to produce a query vector $q_{i} = x_{i, query} W^{Q}$ . The matrix of all query vectors is the query matrix: $Q = X_{query} W^{Q}$ Similarly, we construct the key matrix $K = X_{key} W^{K}$ and the value matrix $V = X_{value} W^{V}$ .

The shapes of $W^{Q}, W^{K}, W^{V}$ depend on the input and head dimensions; these projection matrices need not be square.

Attention weights are calculated using the query and key vectors: the attention weight $a_{i j}$ from token $i$ to token $j$ is the dot product between $q_{i}$ and $k_{j}$ . The attention weights are divided by the square root of the dimension of the key vectors, $\sqrt{d_{k}}$ , which stabilizes gradients during training, and passed through a softmax which normalizes the weights. The fact that $W^{Q}$ and $W^{K}$ are different matrices allows attention to be non-symmetric: if token $i$ attends to token $j$ (i.e. $q_{i} \cdot k_{j}$ is large), this does not necessarily mean that token $j$ will attend to token $i$ (i.e. $q_{j} \cdot k_{i}$ could be small). The output of the attention unit for token $i$ is the weighted sum of the value vectors of all tokens, weighted by $a_{i j}$ , the attention from token $i$ to each token.

The attention calculation for all tokens can be expressed as one large matrix calculation using the softmax function, which is useful for training due to computational matrix operation optimizations that quickly compute matrix operations. The matrices $Q$ , $K$ and $V$ are defined as the matrices where the $i$ th rows are vectors $q_{i}$ , $k_{i}$ , and $v_{i}$ respectively. Then we can represent the attention as $\begin{matrix} Attention (Q, K, V) = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}}) V \end{matrix}$

where the softmax is applied over each of the rows of the matrix.

The number of dimensions in a query vector is query size $d_{query}$ and similarly for the key size $d_{key}$ and value size $d_{value}$ . The output dimension of an attention head is its head dimension $d_{head}$ . The attention mechanism requires the following three equalities to hold: $ℓ_{seq, key} = ℓ_{seq, value}, d_{query} = d_{key}, d_{value} = d_{head}$ but is otherwise unconstrained.

If the attention head is used in a self-attention fashion, then $X_{query} = X_{key} = X_{value}$ . If the attention head is used in a cross-attention fashion, then usually $X_{query} \neq X_{key} = X_{value}$ . It is theoretically possible for all three to be different, but that is rarely the case in practice.

Multihead attention

One set of $(W^{Q}, W^{K}, W^{V})$ matrices is called an attention head, and each layer in a transformer model has multiple attention heads. While each attention head attends to the tokens that are relevant to each token, multiple attention heads allow the model to do this for different definitions of "relevance". Specifically, the query and key projection matrices, $W^{Q}$ and $W^{K}$ , which are involved in the attention score computation, defines the "relevance". Meanwhile, the value projection matrix $W^{V}$ , in combination with the part of the output projection matrix $W^{O}$ , determines how the attended tokens influence what information is passed to subsequent layers and ultimately the output logits. In addition, the scope of attention, or the range of token relationships captured by each attention head, can expand as tokens pass through successive layers. This allows the model to capture more complex and long-range dependencies in deeper layers. Analyses of some models have found attention heads that focus on patterns such as nearby words or syntactic relations; these patterns vary across heads and models. The computations for each attention head can be performed in parallel, which allows for fast processing. The outputs for the attention layer are concatenated to pass into the feedforward neural network layers.

Concretely, let the multiple attention heads be indexed by $i$ , then we have $MultiheadAttention (Q, K, V) = {Concat}_{i \in [n_{heads}]} (Attention (X W^{Q, i}, X W^{K, i}, X W^{V, i})) W^{O}$ where $X$ contains the input representations to the attention layer (embeddings in the first layer and hidden states in later layers), and the matrices $W^{Q, i}, W^{K, i}, W^{V, i}$ are projection matrices for attention head $i$ , and $W^{O}$ is a final projection matrix owned by the whole multihead attention module.

It is theoretically possible for each attention head to have a different head dimension $d_{head}$ , but that is rarely the case in practice.

As an example, in the smallest GPT-2 model, there are only self-attention mechanisms. It has the following dimensions: $d_{emb} = 768, n_{head} = 12, d_{head} = 64$ Since $12 \times 64 = 768$ , its output projection matrix $W^{O} \in ℝ^{(12 \times 64) \times 768}$ is a square matrix.

Masked attention

Autoregressive decoders predict each token from earlier tokens. During training, causal self-attention can process all sequence positions in parallel while preventing position $i$ from attending to later positions $j > i$ . During inference, the model generates tokens autoregressively. This mask is applied before softmax by adding a matrix $M$ with $- \infty$ at masked positions and $0$ elsewhere: $\begin{matrix} MaskedAttention (Q, K, V) = softmax (\frac{Q K^{T}}{\sqrt{d_{k}}} + M) V \end{matrix}$ The following matrix is commonly used in decoder self-attention modules, called "causal masking": $M_{causal} = [\begin{matrix} 0 & - \infty & - \infty & \dots & - \infty \\ 0 & 0 & - \infty & \dots & - \infty \\ 0 & 0 & 0 & \dots & - \infty \\ ⋮ & ⋮ & ⋮ & ⋱ & ⋮ \\ 0 & 0 & 0 & \dots & 0 \end{matrix}]$

In words, it means that each token can pay attention to itself, and every token before it, but not any after it. A non-masked attention module can be thought of as a masked attention module where the mask has all entries zero. As an example of an uncommon use of mask matrix, the XLNet considers all masks of the form $P M_{causal} P^{- 1}$ , where $P$ is a random permutation matrix.

Encoder

An encoder consists of an embedding layer, followed by multiple encoder layers.

Each encoder layer consists of two major components: a self-attention mechanism and a feed-forward layer. It takes an input as a sequence of input vectors, applies the self-attention mechanism, to produce an intermediate sequence of vectors, then applies the feed-forward layer for each vector individually. Schematically, we have: $\begin{matrix} given input vectors & h_{0}, h_{1}, \dots \\ combine them into a matrix H & = [\begin{matrix} h_{0} \\ h_{1} \\ ⋮ \end{matrix}] \\ EncoderLayer (H) & = [\begin{matrix} FFN (MultiheadAttention (H, H, H)_{0}) \\ FFN (MultiheadAttention (H, H, H)_{1}) \\ ⋮ \end{matrix}] \end{matrix}$

where $FFN$ stands for "feed-forward network". We can more succinctly write it as $EncoderLayer (H) = FFN (MultiheadAttention (H, H, H))$ with the implicit convention that the $FFN$ is applied to each row of the matrix individually.

The encoder layers are stacked. The first encoder layer takes the sequence of input vectors from the embedding layer, producing a sequence of vectors. This sequence of vectors is processed by the second encoder, and so on. The output from the final encoder layer is then used by the decoder.

As the encoder processes the entire input all at once, every token can attend to every other token (all-to-all attention), so there is no need for causal masking.

Decoder

A decoder consists of an embedding layer, followed by multiple decoder layers, followed by an un-embedding layer.

Each decoder consists of three major components: a causally masked self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network. The decoder functions in a similar fashion to the encoder, but an additional attention mechanism is inserted which instead draws relevant information from the encodings generated by the encoders. This mechanism can also be called the encoder–decoder attention.^{[non-primary source needed]} Like the first encoder, the first decoder takes positional information and embeddings of the output sequence as its input, rather than encodings. During training, target tokens are shifted right before being supplied to the decoder, and causal masking prevents each position from attending to later target tokens. This allows for autoregressive text generation. For decoding, all-to-all attention is inappropriate, because a token cannot attend to tokens not yet generated. Thus, the self-attention module in the decoder is causally masked.

In contrast, the cross-attention mechanism attends to the output vectors of the encoder, which is computed before the decoder starts decoding. Consequently, there is no need for masking in the cross-attention mechanism.

Schematically, we have: $\begin{matrix} H^{'} & = MaskedMultiheadAttention (H, H, H) \\ DecoderLayer (H) & = FFN (MultiheadAttention (H^{'}, H^{E}, H^{E})) \end{matrix}$ where $H^{E}$ is the matrix with rows being the output vectors from the encoder.

The last decoder is followed by a final un-embedding layer to produce the output probabilities over the vocabulary. Then, one of the tokens is sampled according to the probability, and the decoder can be run again to produce the next token, etc., autoregressively generating output text.

Full transformer architecture

Sublayers

Each encoder layer contains 2 sublayers: the self-attention and the feedforward network. Each decoder layer contains 3 sublayers: the causally masked self-attention, the cross-attention, and the feedforward network.

The original Transformer uses residual connections and layer normalization (LayerNorm, or LN). Their placement and use vary across Transformer architectures.

The residual connections are introduced to avoid vanishing gradient issues and stabilize the training process. They can be expressed by $x \mapsto F (x) + x$ , where $F$ is a given component of the transformer. Adding the input $x$ can preserve the input information and avoid issues when the gradient of $F (x)$ is close to zero.

Similarly to how the feedforward network modules are applied individually to each vector, the LayerNorm is also applied individually to each vector.

There are two common conventions in use: the post-LN and the pre-LN convention. In the post-LN convention, the output of each sublayer is $L a y e r N o r m (x + S u b l a y e r (x))$ where $S u b l a y e r (x)$ is the function implemented by the sublayer itself.

In the pre-LN convention, the output of each sublayer is $x + S u b l a y e r (L a y e r N o r m (x))$ The original 2017 transformer used the post-LN convention. It was difficult to train and required careful hyperparameter tuning and a "warm-up" in learning rate, where it starts small and gradually increases. The pre-LN convention, proposed several times in 2018, was found to be easier to train, requiring no warm-up, leading to faster convergence.

Pseudocode

The following is the pseudocode for a standard pre-LN encoder–decoder transformer, adapted from Formal Algorithms for Transformers.

input: Encoder input t_e
       Decoder input t_d
output: Array of probability distributions, with shape (decoder vocabulary size x length(decoder output sequence))

/* encoder */
z_e ← encoder.tokenizer(t_e)

for each t in 1:length(z_e) do
    z_e[t] ← encoder.embedding(z_e[t]) + encoder.positional_embedding(t)

for each l in 1:length(encoder.layers) do
    layer ← encoder.layers[l]

    /* first sublayer */
    z_e_copy ← copy(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← layer.layer_norm(z_e[t])
    z_e ← layer.multihead_attention(z_e, z_e, z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← z_e[t] + z_e_copy[t]

    /* second sublayer */
    z_e_copy ← copy(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← layer.layer_norm(z_e[t])
    z_e ← layer.feedforward(z_e)
    for each t in 1:length(z_e) do
        z_e[t] ← z_e[t] + z_e_copy[t]

for each t in 1:length(z_e) do
    z_e[t] ← encoder.final_layer_norm(z_e[t])

/* decoder */
z_d ← decoder.tokenizer(t_d)

for each t in 1:length(z_d) do
    z_d[t] ← decoder.embedding(z_d[t]) + decoder.positional_embedding(t)

for each l in 1:length(decoder.layers) do
        layer ← decoder.layers[l]

        /* first sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.masked_multihead_attention(z_d, z_d, z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← z_d[t] + z_d_copy[t]

        /* second sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.multihead_attention(z_d, z_e, z_e) 
       for each t in 1:length(z_d) do
           z_d[t] ← z_d[t] + z_d_copy[t]

        /* third sublayer */
        z_d_copy ← copy(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← layer.layer_norm(z_d[t])
        z_d ← layer.feedforward(z_d)
        for each t in 1:length(z_d) do
            z_d[t] ← z_d[t] + z_d_copy[t]

z_d ← decoder.final_layer_norm(z_d)

output_distributions ← []
for each t in 1:length(z_d) do
    output_distributions.append(decoder.unembed(z_d[t]))

return output_distributions

Terminology

The transformer architecture, being modular, allows variations. Several common variations are described here.

An "encoder-only" Transformer maps an input sequence to contextualized representations. It is commonly used for text classification, retrieval, and representation learning. BERT is an example. Encoder-only models are also called "all-to-all" or "BERT-like" because their self-attention can generally use the full input sequence.

A "decoder-only" Transformer has causal self-attention and feed-forward layers, with no separate encoder or encoder–decoder cross-attention. It is commonly used for text generation and instruction following. The models in the GPT series and Chinchilla series are examples. Such models are also called autoregressive or causal language models.

An "encoder–decoder" Transformer has an encoder that processes an input sequence and a decoder that generates an output sequence. In the original design, each decoder layer includes causal self-attention, cross-attention to encoder representations, and a feed-forward network; later models vary these details. The T5 series is encoder–decoder.

A "prefixLM" (prefix language model) is a decoder-only architecture, but with prefix masking, which is different from causal masking. Specifically, it has mask of the form^{: Figure 3} $M_{prefixLM} = [\begin{matrix} 𝟎 & - \infty \\ 𝟎 & M_{causal} \end{matrix}]$ where the block rows and columns correspond to prefix and continuation positions. Prefix tokens can attend to all prefix tokens; continuation tokens can attend to the prefix and earlier continuation tokens. This mask combines bidirectional attention within the prefix with causal attention over the continuation.

There are also mixed seq2seq models. For example, in 2020, Google Translate replaced the previous RNN-encoder–RNN-decoder model with a transformer-encoder–RNN-decoder model, as transformer-based decoders did not appear to significantly increase quality unlike the encoder, while the RNN decoder was much faster.

Subsequent work

Many improvements have been added to the Transformer architecture. Transformer variants use different combinations of design choices, including positional encodings, normalization methods, and feed-forward modules.

Alternative feedforward module

The original transformer's feedforward module is a standard linear-ReLU-linear network. Other feedforward modules were developed. The Llama series and PaLM used SwiGLU; both GPT-1 and BERT used GELU.

Alternative activation functions are often used in combination with Gated Linear Units in the feedforward module.

Alternative normalizations

The normalization used in the transformer can be different from LayerNorm. One example is RMSNorm which is used in the Llama series. Other examples include ScaleNorm and FixNorm.

Alternative positional encodings

Transformers may use other positional encoding methods than sinusoidal.

The original transformer paper reported using a learned positional encoding, but finding it not superior to the sinusoidal one. Later, found that causal masking itself provides enough signal to a transformer decoder that it can learn to implicitly perform absolute positional encoding without the positional encoding module.

Efficient implementation

The transformer model has been implemented in standard deep learning frameworks such as TensorFlow and PyTorch. Transformers is a library produced by Hugging Face that supplies transformer-based architectures and pretrained models.

KV caching

When an autoregressive transformer is used for inference, such as generating text, the query vector is different at each step, but the already-computed key and value vectors are always the same. The KV caching method saves the computed key and value vectors at each attention block, so that they are not recomputed at each new token. PagedAttention applies memory paging to KV caching.

For repeated requests with the same prefix, an inference system may cache the prefix's key and value states and reuse them, provided the model and relevant settings match. This can reduce repeated computation; the cache consumes memory and may be invalidated when the prompt or model changes.

In general, when a user uses an autoregressive transformer to generate a continuation to a sequence of tokens, the model would first perform a forward-pass on this sequence, whereby the KV caches over this sequence are computed. This is called prefilling. After prefilling, the actual token generation is called decoding. Hyperscalers serving large Transformer models may use disaggregated inference, wherein prefilling and decoding are performed on separately specialized hardware.

Continuous Batching

When using autoregressive transformer models, requests can have different input lengths and can finish generating at different times. Continuous batching adds new requests to batch and removes completed requests between decoding iterations instead of waiting for every request in a batch to finish. This can improve hardware utilization and throughput for concurrent workloads. Its effect on latency depends on the workload and scheduling policy.

Quantization

Transformer models are often quantized for inference, meaning that weights, activations, or both are represented with lower-precision numerical formats than the floating point formats used during training. Quantization can reduce memory use and data transfer costs and can improve inference speed on hardware that supports low precision arithmetic. The trade off is that aggressive quantization can come at the cost of model accuracy or may need calibration and specialized quantization methods to perform well.

FlashAttention

FlashAttention is an exact attention algorithm that tiles the computation to reduce data movement between high-bandwidth memory and on-chip SRAM. It avoids materializing the full attention matrix, reducing memory traffic and often improving speed and memory use on supported GPUs.The FlashAttention method is a communication-avoiding algorithm that fuses these operations into a single loop, increasing the arithmetic intensity. It is an online algorithm that computes the following quantities: $\begin{matrix} z_{i} & = q^{T} k_{i} \\ m_{i} & = max (z_{1}, \dots, z_{i}) & = & max (m_{i - 1}, z_{i}) \\ ℓ_{i} & = e^{z_{1} - m_{i}} + \dots + e^{z_{i} - m_{i}} & = & e^{m_{i - 1} - m_{i}} ℓ_{i - 1} + e^{z_{i} - m_{i}} \\ o_{i} & = e^{z_{1} - m_{i}} v_{1} + \dots + e^{z_{i} - m_{i}} v_{i} & = & e^{m_{i - 1} - m_{i}} o_{i - 1} + e^{z_{i} - m_{i}} v_{i} \end{matrix}$ and returns $o_{N} / ℓ_{N}$ . In practice, FlashAttention operates over multiple queries and keys per loop iteration, in a similar way as blocked matrix multiplication. If backpropagation is needed, then the output vectors and the intermediate arrays $[m_{1}, \dots, m_{N}], [ℓ_{1}, \dots, ℓ_{N}]$ are cached, and during the backward pass, attention matrices are rematerialized from these, making it a form of gradient checkpointing.

FlashAttention-2 improves parallelism and work partitioning, which can increase performance across supported sequence lengths.

Multi-Query Attention

Multi-Query Attention (MQA) changes the Multihead Attention mechanism. Whereas normally,

$MultiheadAttention (Q, K, V) = {Concat}_{i \in [n_{heads}]} (Attention (X W_{i}^{Q}, X W_{i}^{K}, X W_{i}^{V})) W^{O}$ with Multi-Query Attention, there is just one $W^{K}, W^{V}$ , thus:

$MultiQueryAttention (Q, K, V) = {Concat}_{i \in [n_{heads}]} (Attention (X W_{i}^{Q}, X W^{K}, X W^{V})) W^{O}$

Sharing key and value projections across heads reduces the size of the KV cache and memory bandwidth required during autoregressive decoding; reported speed and quality effects depend on the model and task.

More generally, grouped-query attention (GQA) partitions attention heads into groups, each of which shares the key-value pair. MQA is GQA with one group, while standard Multihead Attention is GQA with the maximal number of groups.

Multihead Latent Attention (MLA) is an alternative attention design that compresses key–value representations into a low-dimensional latent vector. The compressed representation reduces the KV-cache memory required during inference; MLA is distinct from MQA, which shares key and value heads across query heads.

Speculative decoding

Speculative decoding accelerates autoregressive generation by using a smaller draft model or another method to propose several tokens, then asking the target model to verify the proposal in a batched forward pass. The system accepts verified draft tokens and obtains a correction token when needed. With sampling, an acceptance and correction procedure can preserve the target model's output distribution. The speedup depends on draft-model cost, acceptance rate, and hardware.

Multi-token prediction (MTP) is a training objective that adds prediction heads for multiple future tokens. It can provide additional training signal, and some systems use MTP predictions to support speculative decoding; MTP is not, by itself, a general inference method that produces an arbitrary number of future tokens in one pass.

Sub-quadratic transformers

Training transformer-based architectures can be expensive, especially for long inputs. Many methods have been developed to attempt to address the issue. In the image domain, Swin transformer is an efficient architecture that performs attention inside shifting windows. In the audio domain, SepTr decouples the attention in time and frequency domains. Long Range Arena (2020) is a standard benchmark for comparing the behavior of transformer architectures over long inputs.

The standard attention graph is either all-to-all or causal, both of which scales as $O (N^{2})$ where $N$ is the number of tokens in a sequence. In both cases, there are $O (N^{2})$ pairs of tokens such that one influences the other within the attention block. By pruning the attention graph, the number of pairs of token influences can drop below $O (N^{2})$ . Such architectures are often referred to as sparse attention, or sparse Transformers.

Sparse attention uses sparse attention patterns to reduce computation for long sequences. BigBird combines local-window, random, and global attention links; with fixed block sizes, this pattern can scale linearly with sequence length.

Multimodality

Transformers can also be used/adapted for modalities (input or output) beyond just text, usually by finding a way to "tokenize" the modality.

Multimodal models can either be trained from scratch, or by finetuning. A 2022 study found that transformers pretrained only on natural language can be finetuned on only 0.03% of parameters and become competitive with LSTMs on a variety of logical and visual tasks, demonstrating transfer learning. LLaVA was a vision-language model composed of a language model (Vicuna-13B) and a vision model (ViT-L/14), connected by a learned projection layer. In its first training stage, the projection layer was trained while the vision encoder and language model were frozen; in the later visual-instruction-tuning stage, the vision encoder remained frozen while the projection layer and language model were fine-tuned.

Vision transformers adapt the transformer to computer vision by breaking down input images as a series of patches, turning them into vectors, and treating them like embedding vectors of tokens in a standard transformer.^{[citation needed]}

Conformer combines self-attention with convolution modules to model both global and local patterns in speech. Whisper converts audio to log-Mel spectrogram features and processes them with convolutional layers and an encoder–decoder Transformer; it does not divide spectrograms into image-style patches.

Perceivers are a variant of transformers designed for multimodality.

For image generation, notable architectures are DALL-E 1 (2021), Parti (2022), Phenaki (2023), and Muse (2023). Unlike later models, DALL-E is not a diffusion model. Instead, it uses a decoder-only transformer that autoregressively generates a text, followed by the token representation of an image, which is then decoded by a discrete variational autoencoder (dVAE) into an image. Parti is an encoder–decoder transformer, where the encoder processes a text prompt, and the decoder generates a token representation of an image. Muse is an encoder-only transformer that is trained to predict masked image tokens from unmasked image tokens. During generation, all input tokens are masked, and the highest-confidence predictions are included for the next iteration, until all tokens are predicted. Phenaki is a text-to-video model. It is a bidirectional masked transformer conditioned on pre-computed text tokens. The generated tokens are then decoded to a video.

Applications

The transformer has had great success in natural language processing (NLP). Many large language models such as GPT-2, GPT-3, GPT-4, Gemini, AlbertAGPT, Claude, BERT, Grok, XLNet, RoBERTa and ChatGPT demonstrate the ability of transformers to perform a wide variety of NLP-related subtasks and their related real-world applications, including:

machine translation
time series prediction
document summarization
document generation
named entity recognition (NER)
writing computer code based on requirements expressed in natural language.
speech-to-text

Beyond traditional NLP, the transformer architecture has had success in other applications, such as:

biological sequence analysis
video understanding
protein folding (such as AlphaFold)
evaluating chess board positions. A transformer-based chess policy achieved an Elo rating of 2,895 in Lichess blitz games against human players without explicit search.
service function chain embedding

Notes

↑ Gated recurrent units (2014) further reduced its complexity.
↑ Some architectures, such as RWKV (Receptance Weighted Key Value) or state space models, avoid the issue.

References

1 2 3 4 5 6 7 8 9 10 Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (2017). "Attention is All You Need" ( (PDF)). 31st Conference on Neural Information Processing Systems (NIPS 2017), 4-9 December 2017, Long Beach, CA, USA. Vol. 30. Curran Associates, Inc. arXiv:1706.03762. ISBN 978-1-5108-6096-4. Archived from the original on 2024-02-21. Retrieved 2023-10-31.
1 2 Hochreiter, Sepp; Schmidhuber, Jürgen (November 1997). "Long Short-Term Memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276.
1 2 Bahdanau; Cho, Kyunghyun; Bengio, Yoshua (September 1, 2014). "Neural Machine Translation by Jointly Learning to Align and Translate". arXiv:1409.0473 [cs.CL].
↑ Luong, Minh-Thang; Pham, Hieu; Manning, Christopher D. (August 17, 2015). "Effective Approaches to Attention-based Neural Machine Translation". arXiv:1508.04025 [cs.CL].
↑ Parisotto, Emilio; Song, Francis; Rae, Jack; Pascanu, Razvan; Gulcehre, Caglar; Jayakumar, Siddhant; Jaderberg, Max; Kaufman, Raphaël Lopez; Clark, Aidan; Noury, Seb; Botvinick, Matthew; Heess, Nicolas; Hadsell, Raia (2020-11-21). "Stabilizing Transformers for Reinforcement Learning". Proceedings of the 37th International Conference on Machine Learning. PMLR: 7487–7498. Archived from the original on 2024-08-09. Retrieved 2024-08-09.
↑ Radford, Alec; Jong Wook Kim; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). "Robust Speech Recognition via Large-Scale Weak Supervision". arXiv:2212.04356 [eess.AS].
↑ Monastirsky, Maxim; Azulay, Osher; Sintov, Avishai (February 2023). "Learning to Throw With a Handful of Samples Using Decision Transformers". IEEE Robotics and Automation Letters. 8 (2): 576–583. Bibcode:2023IRAL....8..576M. doi:10.1109/LRA.2022.3229266.
1 2 Ruoss, Anian; Delétang, Grégoire; Medapati, Sourabh; Grau-Moya, Jordi; Wenliang, Li; Catt, Elliot; Reid, John; Genewein, Tim (2024-02-07). "Grandmaster-Level Chess Without Search". arXiv:2402.04494v1 [cs.LG].
1 2 Wolf, Thomas; Debut, Lysandre; Sanh, Victor; Chaumond, Julien; Delangue, Clement; Moi, Anthony; Cistac, Pierric; Rault, Tim; Louf, Remi; Funtowicz, Morgan; Davison, Joe; Shleifer, Sam; von Platen, Patrick; Ma, Clara; Jernite, Yacine; Plu, Julien; Xu, Canwen; Le Scao, Teven; Gugger, Sylvain; Drame, Mariama; Lhoest, Quentin; Rush, Alexander (2020). "Transformers: State-of-the-Art Natural Language Processing". Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. doi:10.18653/v1/2020.emnlp-demos.6.
↑ Feldman, J; Ballard, D (September 1982). "Connectionist models and their properties". Cognitive Science. 6 (3): 205–254. doi:10.1016/S0364-0213(82)80001-3.
↑ Rumelhart, David E.; Hinton, Geoffrey E.; McClelland, James L. (1986). "Interactive Processes in Speech Perception". Parallel Distributed Processing. doi:10.7551/mitpress/5236.003.0005. ISBN 978-0-262-29140-8.
↑ Giles, C. Lee; Maxwell, Tom (December 1987). "Learning, invariance, and generalization in high-order neural networks". Applied Optics. 26 (23): 4972–4978. doi:10.1364/AO.26.004972. PMID 20523475.
1 2 Cho, Kyunghyun; van Merriënboer, Bart; Gulcehre, Caglar; Bahdanau, Dzmitry; Bougares, Fethi; Schwenk, Holger; Bengio, Yoshua (October 2014). "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation". In Moschitti, Alessandro; Pang, Bo; Daelemans, Walter (eds.). Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics. pp. 1724–1734. arXiv:1406.1078. doi:10.3115/v1/D14-1179. Archived from the original on 2024-09-06. Retrieved 2024-08-09.
1 2 Sutskever, Ilya; Vinyals, Oriol; Le, Quoc Viet (14 Dec 2014). "Sequence to sequence learning with neural networks". arXiv:1409.3215 [cs.CL]. [first version posted to arXiv on 10 Sep 2014]
↑ Chung, Junyoung; Gulcehre, Caglar; Cho, KyungHyun; Bengio, Yoshua (2014). "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling". arXiv:1412.3555 [cs.NE].
↑ Gruber, Nicole; Jockisch, Alfred (30 June 2020). "Are GRU Cells More Specific and LSTM Cells More Sensitive in Motive Classification of Text?". Frontiers in Artificial Intelligence. 3 40. doi:10.3389/frai.2020.00040. PMC 7861254. PMID 33733157.
↑ Sutskever, Ilya; Vinyals, Oriol; Le, Quoc V (2014). "Sequence to Sequence Learning with Neural Networks". Advances in Neural Information Processing Systems. 27. Curran Associates, Inc. arXiv:1409.3215. Archived from the original on 2025-01-27. Retrieved 2024-07-23.
↑ Luong, Minh-Thang; Pham, Hieu; Manning, Christopher D. (2015). "Effective Approaches to Attention-based Neural Machine Translation". arXiv:1508.04025 [cs.CL].
↑ Wu, Yonghui; et al. (2016-09-01). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". arXiv:1609.08144 [cs.CL].
↑ Lewis-Kraus, Gideon (2016-12-14). "The Great A.I. Awakening". The New York Times. Archived from the original on 24 May 2023. Retrieved 2023-06-22.
↑ Parikh, Ankur P.; Täckström, Oscar; Das, Dipanjan; Uszkoreit, Jakob (2016-09-25). "A Decomposable Attention Model for Natural Language Inference". arXiv:1606.01933 [cs.CL].
1 2 Levy, Steven. "8 Google Employees Invented Modern AI. Here's the Inside Story". Wired. Archived from the original on 20 Mar 2024. Retrieved 2024-08-06.
↑ Cheng, Jianpeng; Dong, Li; Lapata, Mirella (November 2016). "Long Short-Term Memory-Networks for Machine Reading". In Su, Jian; Duh, Kevin; Carreras, Xavier (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin, Texas: Association for Computational Linguistics. pp. 551–561. doi:10.18653/v1/D16-1053. Archived from the original on 2024-08-27. Retrieved 2024-08-27.
↑ Peng, Bo; Alcaide, Eric; Anthony, Quentin; Albalak, Alon; Arcadinho, Samuel; Biderman, Stella; Cao, Huanqi; Cheng, Xin; Chung, Michael (2023-12-10). "RWKV: Reinventing RNNs for the transformer Era". arXiv:2305.13048 [cs.CL].
↑ Marche, Stephen (2024-08-23). "Was Linguistic A.I. Created by Accident?". The New Yorker. Retrieved 2024-08-27.
↑ Vaswani, Ashish; Bengio, Samy; Brevdo, Eugene; Chollet, Francois; Gomez, Aidan; Gouws, Stephan; Jones, Llion; Kaiser, Łukasz; Kalchbrenner, Nal; Parmar, Niki; Sepassi, Ryan; Shazeer, Noam; Uszkoreit, Jakob (March 2018). Cherry, Colin; Neubig, Graham (eds.). "Tensor2Tensor for Neural Machine Translation". Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track). Boston, MA: Association for Machine Translation in the Americas: 193–199. Archived from the original on 2026-04-29. Retrieved 2026-03-31.
↑ Kaiser, Łukasz (2017-06-19). "Accelerating Deep Learning Research with the Tensor2Tensor Library". Google Research Blog. Archived from the original on 2026-04-15. Retrieved 2026-03-31.
1 2 3 4 5 Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 October 2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv:1810.04805v2 [cs.CL].
↑ "Google: BERT now used on almost every English query". Search Engine Land. 2020-10-15. Archived from the original on 2022-05-06. Retrieved 2020-11-24.
1 2 Caswell, Isaac; Liang, Bowen (June 8, 2020). "Recent Advances in Google Translate". Google Research. Archived from the original on 4 Jul 2024. Retrieved 2024-08-07.
↑ "The inside story of how ChatGPT was built from the people who made it". MIT Technology Review. Archived from the original on 2023-03-03. Retrieved 2024-08-06.
↑ "Introducing ChatGPT". OpenAI. 2022-11-30. Archived from the original on 2026-04-05. Retrieved 2026-05-16.
↑ Deign, Jason (December 6, 2024). "How ChatGPT changed… well, almost everything". Cisco Newsroom. Retrieved 2026-10-22.
↑ Kaiser, Lukasz; Gomez, Aidan N.; Shazeer, Noam; Vaswani, Ashish; Parmar, Niki; Jones, Llion; Uszkoreit, Jakob (2017-06-16). "One Model To Learn Them All". arXiv:1706.05137v1 [cs.LG].
↑ Liu, Zhuang; Mao, Hanzi; Wu, Chao-Yuan; Feichtenhofer, Christoph; Darrell, Trevor; Xie, Saining (2022). A ConvNet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11976–11986.
1 2 Xiong, Ruibin; Yang, Yunchang; He, Di; Zheng, Kai; Zheng, Shuxin; Xing, Chen; Zhang, Huishuai; Lan, Yanyan; Wang, Liwei; Liu, Tie-Yan (2020-06-29). "On Layer Normalization in the Transformer Architecture". arXiv:2002.04745 [cs.LG].
↑ Vaswani, Ashish; Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion; Gomez, Aidan N; Kaiser, Ł ukasz; Polosukhin, Illia (2017). "Attention is All you Need". Advances in Neural Information Processing Systems. 30. Curran Associates, Inc.
↑ Narayanan, Deepak; Harlap, Aaron; Phanishayee, Amar; Seshadri, Vivek; Devanur, Nikhil R.; Ganger, Gregory R.; Gibbons, Phillip B.; Zaharia, Matei (2019). "PipeDream: Generalized pipeline parallelism for DNN training". Proceedings of the 27th ACM Symposium on Operating Systems Principles. pp. 1–15. doi:10.1145/3341301.3359646. ISBN 978-1-4503-6873-5.
1 2 3 "Causal language modeling". huggingface.co. Archived from the original on 2023-10-10. Retrieved 2023-10-05.
1 2 3 "Masked language modeling". huggingface.co. Archived from the original on 2023-10-10. Retrieved 2023-10-05.
1 2 3 4 5 Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michael; Zhou, Yanqi; Li, Wei; Liu, Peter J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer". Journal of Machine Learning Research. 21 (140): 1–67. Archived from the original on 2026-04-10. Retrieved 2026-04-18.
↑ Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michael; Zhou, Yanqi; Li, Wei; Liu, Peter J. (2019). "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer". arXiv:1910.10683 [cs.LG].
1 2 Tay, Yi; Dehghani, Mostafa; Tran, Vinh Q.; Garcia, Xavier; Wei, Jason; Wang, Xuezhi; Chung, Hyung Won; Shakeri, Siamak; Bahri, Dara (2023-02-28). "UL2: Unifying Language Learning Paradigms". arXiv:2205.05131 [cs.CL].
↑ Sennrich, Rico; Haddow, Barry; Birch, Alexandra (2016). "Neural Machine Translation of Rare Words with Subword Units". Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/P16-1162.
↑ Kudo, Taku (2018). "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates". Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/D18-2012.
↑ Team, Keras. "Keras documentation: GPT2Backbone model". keras.io. Archived from the original on 2024-08-08. Retrieved 2024-08-08.
↑ Clark, Kevin; Khandelwal, Urvashi; Levy, Omer; Manning, Christopher D. (August 2019). "What Does BERT Look at? An Analysis of BERT's Attention". Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Florence, Italy: Association for Computational Linguistics: 276–286. arXiv:1906.04341. doi:10.18653/v1/W19-4828. Archived from the original on 2020-10-21. Retrieved 2020-05-20.
↑ Yang, Zhilin; Dai, Zihang; Yang, Yiming; Carbonell, Jaime; Salakhutdinov, Russ R; Le, Quoc V (2019). "XLNet: Generalized Autoregressive Pretraining for Language Understanding". Advances in Neural Information Processing Systems. 32. Curran Associates, Inc. arXiv:1906.08237. Archived from the original on 2024-08-09. Retrieved 2024-08-09.
↑ Wang, Qiang; Li, Bei; Xiao, Tong; Zhu, Jingbo; Li, Changliang; Wong, Derek F.; Chao, Lidia S. (2019-06-04). "Learning Deep Transformer Models for Machine Translation". arXiv:1906.01787 [cs.CL].
↑ Phuong, Mary; Hutter, Marcus (2022-07-19). "Formal Algorithms for Transformers". arXiv:2207.09238 [cs.LG].
↑ Dong, Li; Yang, Nan; Wang, Wenhui; Wei, Furu; Liu, Xiaodong; Wang, Yu; Gao, Jianfeng; Zhou, Ming; Hon, Hsiao-Wuen (2019). "Unified Language Model Pre-training for Natural Language Understanding and Generation". Advances in Neural Information Processing Systems. 32. Curran Associates, Inc.
1 2 Shazeer, Noam (2020-02-01). "GLU Variants Improve Transformer". arXiv:2002.05202 [cs.LG].
↑ Hendrycks, Dan; Gimpel, Kevin (2016-06-27). "Gaussian Error Linear Units (GELUs)". arXiv:1606.08415v5 [cs.LG].
↑ Zhang, Biao; Sennrich, Rico (2019). "Root Mean Square Layer Normalization". Advances in Neural Information Processing Systems. 32. Curran Associates, Inc. arXiv:1910.07467. Archived from the original on 2024-09-17. Retrieved 2024-08-09.
1 2 Nguyen, Toan Q.; Salazar, Julian (2019-11-02). Niehues, Jan; Cattoni, Rolando; Stüker, Sebastian; Negri, Matteo; Turchi, Marco; Ha, Thanh-Le; Salesky, Elizabeth; Sanabria, Ramon; Barrault, Loic (eds.). "Transformers without Tears: Improving the Normalization of Self-Attention". Proceedings of the 16th International Conference on Spoken Language Translation. Hong Kong: Association for Computational Linguistics. arXiv:1910.05895. doi:10.5281/zenodo.3525484. Archived from the original on 2024-08-09. Retrieved 2024-08-09.
↑ Dufter, Philipp; Schmitt, Martin; Schütze, Hinrich (September 2022). "Position Information in Transformers: An Overview". Computational Linguistics. 48 (3): 733–763. arXiv:2102.11090. doi:10.1162/coli_a_00445.
↑ Gehring, Jonas; Auli, Michael; Grangier, David; Yarats, Denis; Dauphin, Yann N. (2017-07-17). "Convolutional Sequence to Sequence Learning". Proceedings of the 34th International Conference on Machine Learning. PMLR: 1243–1252. Archived from the original on 2024-08-09. Retrieved 2024-08-09.
↑ Haviv, Adi; Ram, Ori; Press, Ofir; Izsak, Peter; Levy, Omer (2022-12-05). "Transformer Language Models without Positional Encodings Still Learn Positional Information". arXiv:2203.16634 [cs.CL].
↑ Kwon, Woosuk; Li, Zhuohan; Zhuang, Siyuan; Sheng, Ying; Zheng, Lianmin; Yu, Cody Hao; Gonzalez, Joseph; Zhang, Hao; Stoica, Ion (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention". Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626. doi:10.1145/3600006.3613165. ISBN 979-8-4007-0229-7.
↑ "vllm-project/vllm". vLLM. 2024-06-20. Archived from the original on 2024-06-18. Retrieved 2024-06-20.
↑ Zhuohan Li, Woosuk Kwon; Zhuang, Siyuan; Sheng, Ying; Zheng, Lianmin; Yu, Cody; Gonzalez, Joey; Zhang, Hao; Stoica, Ion (2023-06-20). "vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention". vLLM Blog. Archived from the original on 2024-06-20. Retrieved 2024-06-20.
↑ Hu, Cunchen; Huang, Heyang; Xu, Liangliang; Chen, Xusheng; Xu, Jiang; Chen, Shuang; Feng, Hao; Wang, Chenxi; Wang, Sa (2024-01-20). "Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads". arXiv:2401.11181 [cs.DC].
↑ Yu, Gyeong-In; Jeong, Joo Seong; Kim, Geon-Woo; Kim, Soojeong; Chun, Byung-Gon (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. pp. 521–538. ISBN 978-1-939133-28-1.
↑ Dettmers, Tim; Lewis, Mike; Belkada, Younes; Zettlemoyer, Luke (2022-08-15). "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale". arXiv:2208.07339v2 [cs.LG].
1 2 Dao, Tri; Ermon, Stefano; Fu, Dan; Ré, Christopher; Rudra, Atri (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". Advances in Neural Information Processing Systems 35. pp. 16344–16359. arXiv:2205.14135. doi:10.52202/068431-1189. ISBN 978-1-7138-7108-8.
↑ Milakov, Maxim; Gimelshein, Natalia (2018). "Online normalizer calculation for softmax". arXiv:1805.02867 [cs.PF].
↑ "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning". crfm.stanford.edu. Archived from the original on 2023-07-18. Retrieved 2023-07-18.
↑ Chowdhery, Aakanksha; Narang, Sharan; Devlin, Jacob; Bosma, Maarten; Mishra, Gaurav; Roberts, Adam; Barham, Paul; Chung, Hyung Won; Sutton, Charles; Gehrmann, Sebastian; Schuh, Parker; Shi, Kensen; Tsvyashchenko, Sasha; Maynez, Joshua; Rao, Abhishek (2022-04-01). "PaLM: Scaling Language Modeling with Pathways". arXiv:2204.02311 [cs.CL].
↑ Ainslie, Joshua; Lee-Thorp, James; de Jong, Michiel; Zemlyanskiy, Yury; Lebrón, Federico; Sanghai, Sumit (2023-12-23). "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints". arXiv:2305.13245 [cs.CL].
1 2 DeepSeek-AI; Liu, Aixin; Feng, Bei; Wang, Bin; Wang, Bingxuan; Liu, Bo; Zhao, Chenggang; Dengr, Chengqi; Ruan, Chong (19 June 2024). "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model". arXiv:2405.04434 [cs.CL]..
↑ Leviathan, Yaniv; Kalman, Matan; Matias, Yossi (2023-05-18). "Fast Inference from Transformers via Speculative Decoding". arXiv:2211.17192 [cs.LG].
↑ Chen, Charlie; Borgeaud, Sebastian; Irving, Geoffrey; Lespiau, Jean-Baptiste; Sifre, Laurent; Jumper, John (2023-02-02). "Accelerating Large Language Model Decoding with Speculative Sampling". arXiv:2302.01318 [cs.CL].
↑ Gloeckle, Fabian; Badr Youbi Idrissi; Rozière, Baptiste; Lopez-Paz, David; Synnaeve, Gabriel (2024). "Better & Faster Large Language Models via Multi-token Prediction". arXiv:2404.19737 [cs.CL].
↑ DeepSeek-AI; et al. (2024). "DeepSeek-V3 Technical Report". arXiv:2412.19437 [cs.CL].
↑ Kitaev, Nikita; Kaiser, Łukasz; Levskaya, Anselm (2020). "Reformer: The Efficient Transformer". arXiv:2001.04451 [cs.LG].
↑ Liu, Ze; Lin, Yutong; Cao, Yue; Hu, Han; Wei, Yixuan; Zhang, Zheng; Lin, Stephen; Guo, Baining (2021). "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows". 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE. pp. 9992–10002. arXiv:2103.14030. doi:10.1109/ICCV48922.2021.00986. ISBN 978-1-6654-2812-5.
↑ Ristea, Nicolaea Catalin; Ionescu, Radu Tudor; Khan, Fahad Shahbaz (2022-09-18). "SepTr: Separable Transformer for Audio Spectrogram Processing". Interspeech. ISCA: 4103–4107. arXiv:2203.09581. doi:10.21437/Interspeech.2022-249.
↑ Tay, Yi; Dehghani, Mostafa; Abnar, Samira; Shen, Yikang; Bahri, Dara; Pham, Philip; Rao, Jinfeng; Yang, Liu; Ruder, Sebastian; Metzler, Donald (2020-11-08). "Long Range Arena: A Benchmark for Efficient Transformers". arXiv:2011.04006 [cs.LG].
↑ Child, Rewon; Gray, Scott; Radford, Alec; Sutskever, Ilya (2019-04-23). "Generating Long Sequences with Sparse Transformers". arXiv:1904.10509 [cs.LG].
↑ Zaheer, Manzil; Gupta, Guru; Kale, Shankar; Kumar, Yash; Contractor, Danish; Rathod, Vishwas; Re, Christopher (2020-10-06). "Big Bird: Transformers for Longer Sequences". arXiv:2007.14062 [cs.CL].
↑ Lu, Kevin; Grover, Aditya; Abbeel, Pieter; Mordatch, Igor (2022). "Frozen Pretrained Transformers as Universal Computation Engines". Proceedings of the AAAI Conference on Artificial Intelligence. 36 (7): 7628–7636. doi:10.1609/aaai.v36i7.20729.
↑ "Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality | LMSYS Org". lmsys.org. 30 March 2023. Archived from the original on 2024-08-12. Retrieved 2024-08-11.
↑ Liu, Haotian; Li, Chunyuan; Wu, Qingyang; Lee, Yong Jae (2023-12-15). "Visual Instruction Tuning". Advances in Neural Information Processing Systems. Vol. 36. pp. 34892–34916. doi:10.52202/075280-1516. ISBN 978-1-7138-9911-2. Archived from the original on 2024-09-26. Retrieved 2024-08-11.
↑ "Conformer: Convolution-augmented Transformer for Speech Recognition". arXiv:2005.08100 [cs.CL]. A bot will complete this citation soon. Click here to jump the queue
↑ Radford, Alec; Kim, Jong Wook; Xu, Tao; Brockman, Greg; McLeavey, Christine; Sutskever, Ilya (2022). "Robust Speech Recognition via Large-Scale Weak Supervision". arXiv:2212.04356 [eess.AS].
↑ Jaegle, Andrew; Gimeno, Felix; Brock, Andrew; Zisserman, Andrew; Vinyals, Oriol; Carreira, Joao (2021-06-22). "Perceiver: General Perception with Iterative Attention". arXiv:2103.03206 [cs.CV].
↑ Jaegle, Andrew; Borgeaud, Sebastian; Alayrac, Jean-Baptiste; Doersch, Carl; Ionescu, Catalin; Ding, David; Koppula, Skanda; Zoran, Daniel; Brock, Andrew; Shelhamer, Evan; Hénaff, Olivier (2021-08-02). "Perceiver IO: A General Architecture for Structured Inputs & Outputs". arXiv:2107.14795 [cs.LG].
↑ "Parti: Pathways Autoregressive Text-to-Image Model". sites.research.google. Archived from the original on 2024-08-10. Retrieved 2024-08-09.
1 2 Villegas, Ruben; Babaeizadeh, Mohammad; Kindermans, Pieter-Jan; Moraldo, Hernan; Zhang, Han; Saffar, Mohammad Taghi; Castro, Santiago; Kunze, Julius; Erhan, Dumitru (2022-09-29). "Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions". arXiv:2210.02399 [cs.CV].
1 2 Chang, Huiwen; Zhang, Han; Barber, Jarred; Maschinot, A. J.; Lezama, Jose; Jiang, Lu; Yang, Ming-Hsuan; Murphy, Kevin; Freeman, William T. (2023-01-02). "Muse: Text-To-Image Generation via Masked Generative Transformers". arXiv:2301.00704 [cs.CV].
↑ Ramesh, Aditya; Pavlov, Mikhail; Goh, Gabriel; Gray, Scott; Voss, Chelsea; Radford, Alec; Chen, Mark; Sutskever, Ilya (2021-02-26). "Zero-Shot Text-to-Image Generation". arXiv:2102.12092 [cs.CV].
↑ Yu, Jiahui; Xu, Yuanzhong; Koh, Jing Yu; Luong, Thang; Baid, Gunjan; Wang, Zirui; Vasudevan, Vijay; Ku, Alexander; Yang, Yinfei (2022-06-21). "Scaling Autoregressive Models for Content-Rich Text-to-Image Generation". arXiv:2206.10789 [cs.CV].
↑ Kariampuzha, William; Alyea, Gioconda; Qu, Sue; Sanjak, Jaleal; Mathé, Ewy; Sid, Eric; Chatelaine, Haley; Yadaw, Arjun; Xu, Yanji; Zhu, Qian (2023). "Precision information extraction for rare disease epidemiology at scale". Journal of Translational Medicine. 21 (1): 157. doi:10.1186/s12967-023-04011-y. PMC 9972634. PMID 36855134.
↑ Hsu, Cyril Shih-Huan; Dalgkitsis, Anestis; Grosso, Paola; Papagianni, Chrysa (2026). "Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioning". IEEE Transactions on Network Science and Engineering. 13: 9247–9264. arXiv:2504.18902. Bibcode:2026ITNSE..13.9247H. doi:10.1109/TNSE.2026.3689920.

v t e Large language models (LLMs)
List of LLMs AI Companies Benchmarks List of chatbots Foundation model Generative AI
Concepts	Language model NLP NLG Computational linguistics Foundation model Small language model Reasoning model Generative pre-trained transformer Transformer Attention KV cache Context window Tokenization Word embedding Parameter Hyperparameter Autoregression Mixture of experts (MoE) Inference Model compression Knowledge distillation Positional encoding Speculative decoding PagedAttention Neural scaling law Multimodality Open weights
Training, prompting, and alignment	Self-supervised learning Supervised learning Fine-tuning Instruction tuning RLHF Constitutional AI AI alignment AI safety Mechanistic interpretability Prompt engineering In-context learning Chain-of-thought prompting RAG Prompt injection Adversarial machine learning Hallucination Stochastic parrot Glitch token Generative engine optimization
Models	Apertus BERT BLOOM Chinchilla Claude DBRX DeepSeek-LLM Gemini Gemma GloVe GLM GPT GPT-J PanGu Granite Inkling Jais LaMDA Laguna S Llama MiMo Minerva Mistral and Mixtral Muse Spark Muse Glimmer Nemotron PaLM Phi Qwen Seq2seq T5 Vicuna Word2vec XLNet
Chatbots and assistants	Amazon Q ChatGPT Character.ai Claude Claude Code DeepSeek Doubao Ernie Bot Gemini Grok Kimi Kimi Code Lumo Microsoft Copilot Meta AI Perplexity AI Sparrow You.com
Agents, coding, and applications	AI agent Intelligent agent AutoGPT LangChain Manus Model Context Protocol Agent2Agent OpenAI Codex Vibe coding Code generation Question answering Machine translation Text summarization Chatbot Virtual assistant
Software	PyTorch TensorFlow Hugging Face llama.cpp LM Studio Ollama vLLM ONNX OpenVINO Vector database ChromaDB Deep learning software Open-source AI software
Hardware and infrastructure	AI data center CUDA GPU High Bandwidth Memory Neural processing unit TPU
Benchmarks, evaluation, and detection	Language model benchmark AI agent benchmark MMLU Humanity's Last Exam LMArena LLM-as-a-Judge Perplexity metric GPTZero Artificial intelligence content detection Undetectable.ai
Datasets and data	Data set Text corpus Common Crawl The Pile Web scraping Synthetic data Training, validation, and test data sets
Organizations	AI21 Labs Alibaba Anthropic Baidu Cohere DeepSeek EleutherAI Google DeepMind Hugging Face Meta AI Microsoft AI Mistral AI MiniMax Moonshot AI Nvidia OpenAI OpenRouter Sarvam AI StepFun Technology Innovation Institute Thinking Machines Lab xAI
People	Sam Altman Dario Amodei Yoshua Bengio Aidan Gomez Demis Hassabis Geoffrey Hinton Andrej Karpathy Yann LeCun Percy Liang Liang Wenfeng Christopher D. Manning Arthur Mensch Mira Murati Alec Radford Noam Shazeer Ilya Sutskever Ashish Vaswani Andrew Ng
Social, economic, and governance	AI anthropomorphism AI arms race AI boom AI bubble AI-induced psychosis AI slop Competition Copyright Deaths linked to chatbots Dependency (GAID) Environmental impact Regulation Ethics Existential risk In education In healthcare Workplace impact
Category:Large language models