DeepSeek (AI)

MoE

DeepSeek-MoE models (Base and Chat), each have 16B parameters (2.7B activated per token, 4K context length). The training was essentially the same as DeepSeek-LLM 7B, and was trained on a part of its training dataset. They claimed performance comparable to a 16B MoE as a 7B non-MoE. It is a variant of the standard sparsely-gated MoE, with "shared experts" that are always queried, and "routed experts" that might not be. They found this to help with expert balancing. In standard MoE, some experts can become overused, while others are rarely used, wasting space. Attempting to balance expert usage causes experts to replicate the same capacity. They proposed the shared experts to learn core capacities that are often used, and let the routed experts learn peripheral capacities that are rarely used.

Math

DeepSeek-Math includes 3 models: Base, Instruct, and RL. Math was trained as follows:

1. Initialize with a previously pretrained DeepSeek-Coder Base v1.5 7B.
2. Further pretrain with 500B tokens (6% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code, 10% Common Crawl). This produced Base.
3. Train an instruction-following model by SFT Base with 776K math problems and tool-use-integrated step-by-step solutions. This produced Instruct.
4. Reinforcement learning (RL): The reward model was a process reward model (PRM) trained from Base according to the Math-Shepherd method. This reward model was then used to train Instruct using Group Relative Policy Optimization (GRPO) on a dataset of 144K math questions "related to GSM8K and MATH". The reward model was continuously updated during training to avoid reward hacking. This resulted in RL.

mHC

In normal residual connections, the residual stream is of form ${\displaystyle x^{\ell +1}=x^{\ell }+f^{\ell }(x^{\ell })}$. mHC divides its residual stream into ${\displaystyle k}$ parts ${\displaystyle x_{1}^{\ell },\dots ,x_{k}^{\ell }}$. The residual stream is updated by$${\displaystyle x_{i}^{\ell +1}=\sum _{j}\underbrace {A_{ij}^{\ell }} _{\text{residual}}x_{j}^{\ell }+\underbrace {C_{i}^{\ell }} _{\text{output}}f^{\ell }\left(\sum _{j}\underbrace {B_{j}^{\ell }} _{\text{input}}x_{j}^{\ell }\right)}$$where

• ${\displaystyle A^{\ell }}$ is a ${\displaystyle k\times k}$ doubly stochastic matrix called the residual mapping. It mixes the ${\displaystyle k}$ parts of the residual stream. It depends on ${\displaystyle x^{\ell }}$:$${\displaystyle A=\operatorname {SK} (\alpha \cdot \operatorname {reshape} _{(k,k)}(W_{k^{2}\times kd}\operatorname {reshape} _{(kd,1)}(x))+W'_{k\times k})}$$here SK denotes the Sinkhorn–Knopp algorithm, ${\displaystyle W}$ denotes learnable matrices of the appropriate sizes, and ${\displaystyle \alpha }$ denotes a learnable scalar.
• ${\displaystyle B^{\ell },C^{\ell }}$ are ${\displaystyle k\times 1}$ vectors with nonnegative entries called the input mapping and output mapping. The input mapping projects the residual stream down by a factor of ${\displaystyle k}$ so that it can be used by the layer, and the output mapping does the opposite. They depend on ${\displaystyle x^{\ell }}$:$${\displaystyle B=\sigma (\alpha '\cdot W''_{k\times kd}\operatorname {reshape} _{(kd,1)}(x)+W'''_{k\times 1}),\qquad C=2\sigma (\alpha ''\cdot W''''_{k\times kd}\operatorname {reshape} _{(kd,1)}(x)+W'''''_{k\times 1})}$$here ${\displaystyle \sigma }$ denotes the logistic function.

KV cache data movement minimization

Standard query-key-value attention retrieves the key-value vectors, uses the query vectors to match the key vectors, and sums the value vectors weighed according to how well their keys matched the query. Since the retrieval step is bandwidth-bound, they used the rerank method in information retrieval to cut down the data movement cost:

• Retrieve all cheap indexing vectors, and match them against the query vector. Find the top-k best matching ones.
• Retrieve the key-value vectors corresponding to these, and then do the query-key-value attention on those only.

They called this the "indexer" method.

In detail, during prefilling, for each KV cache vector ${\displaystyle v_{t}^{KV,(\ell )}}$, the model also computes ${\displaystyle v_{t}^{{\text{index}},(\ell )}=v_{t}^{KV,(\ell )}W^{\text{index}}}$. When decoding a token ${\displaystyle x_{N+1}^{(\ell )}}$, instead of paying attention to all ${\displaystyle v_{t}^{KV,(\ell )}}$, the model reranks:

• Retrieve all ${\displaystyle v_{t}^{{\text{index}},(\ell )}}$ from memory. Compute all ${\displaystyle (x_{N+1}^{(\ell )})^{T}v_{t}^{{\text{index}},(\ell )}}$.
• Retrieve only those ${\displaystyle v_{t}^{KV,(\ell )}}$ for which ${\displaystyle t\in \operatorname {TopK} (k,(x_{N+1}^{(\ell )})^{T}v_{t}^{{\text{index}},(\ell )})}$, and perform the standard attention mechanism on those.

This increases the KV cache size, but decreases the data movement cost.

Each indexer run still takes ${\displaystyle O(N)}$ in data movement, so to cut down the data movement cost further, most layers do not even run the indexers. They simply copy whatever part of the global KV cache was read by the previous layer. They called these layers the "Reuse mode" layers.

To cut down the data movement cost even further, layers 21 to 40 use reranking twice:

• Layer 21 runs the indexer normally, but it produces two lists: a top-${\displaystyle k}$ list and a top-${\displaystyle k'}$ list. It retrieves the top-${\displaystyle k}$ entries for its own attention. They called this the "Full mode".
• Layers 25, 29, 33, 37 run the indexer only over the top-${\displaystyle k'}$ list, each picking its own top-${\displaystyle k}$. They called this the "Reindex mode". Each Reindex mode layer saves data movement compared to a Full mode layer, since it only reads ${\displaystyle k'}$ indexing vectors, instead of ${\displaystyle N}$ indexing vectors.
• Each other layer reuses the top-${\displaystyle k}$ of the layer before it. They called this the "Reuse mode".

They called this the "Hierarchical Sparse Indexer" method, "hierarchical" because it reranks twice.

KV cache compression

Let ${\displaystyle x_{t}^{(\ell )}}$ be the output of the ${\displaystyle \ell }$-th layer at the ${\displaystyle t}$-th token index. In normal multi-headed attention, the KV cache of the ${\displaystyle (\ell +1)}$-th layer has one key vector ${\displaystyle W^{K,(\ell ,h)}x_{t}^{(\ell )}}$ and one value vector ${\displaystyle W^{V,(\ell ,h)}x_{t}^{(\ell )}}$, per (token t, layer l, attention head h). In MLA, the KV cache has one key-value vector ${\displaystyle W^{KV,(\ell )}x_{t}^{(\ell )}}$ per (token t, layer l). This already cuts down the KV cache by ${\displaystyle {\frac {d_{K}+d_{V}}{d_{KV}}}\times {\text{(attention head)}}}$.

To cut down the KV cache even further, they used "token-level compression". That is, they divided the tokens into consecutive blocks of length m, and provide one key-value vector per (token block, layer). This cuts down the KV cache by m. Specifically, suppose that tokens ${\displaystyle t-m+1,\dots ,t}$ are to be merged, then the merged key-value vector is a component-wise weighted sum, where the weight vectors are computed by component-wise softmax:$${\displaystyle {\begin{aligned}v_{t/m}^{KV,(\ell +1)}&=\sum _{i=1}^{m}w_{t-m+i}\odot \;W^{KV,(\ell )}x_{t-m+i}^{(\ell )},\\w_{t-m+i}&={\frac {e^{z_{t-m+i}}}{\sum _{j=1}^{m}e^{z_{t-m+j}}}},\\z_{t-m+i}&=W^{Z,(\ell )}x_{t-m+i}^{(\ell )}\end{aligned}}}$$where ${\displaystyle \odot }$ in the first equation is component-wise multiplication, and exponentiation and division are done component-wise in the second equation.

The actual V4 paper named two attention mechanisms: "Compressed Sparse Attention" (CSA) and "Heavily Compressed Attention" (HCA), which are just the same attention mechanism described in this section, except that one uses ${\displaystyle m=4,k=512}$ (CSA) and the other uses ${\displaystyle m=128,k=\infty }$ (HCA, does not use the indexer).

Multimodality

V4.1 is multimodal, accepting inputs that have images and text interspersed together. Text is tokenized and embedded into a sequence of embedding vectors in the same way of a standard LLM. The images are converted by a vision transformer DeepSeek-ViT into a sequence of image embedding vectors, then these are mapped by a small 2-layer MLP to embedding vectors that DeepSeek-V4.1-Flash accepts (since DeepSeek-ViT and DeepSeek-V4.1-Flash uses different embedding dimensions).

DeepSeek-ViT is a standard ViT with Transformer++ and 2D RoPE. Each image embedding vector it outputs corresponds to a 52×52 pixel image patch.

Training

The training recipe:

• DeepSeek-ViT was trained by SigLIP-style contrastive learning on about 47B image-text pairs.
• It was connected to a 4B MoE LLM and they were jointly pretrained by next-token prediction on 236B tokens of captions, charts, and OCR. The small LLM was discarded after this.
• DeepSeek-ViT was frozen, and connected to DeepSeek-V4.1-Flash. They were pretrained on 45T multimodal tokens by next-token prediction on the text tokens (the image tokens cannot be predicted, since DeepSeek-V4.1-Flash does not output image tokens). The sequence length goes from 64K to 1M at the 34T mark without special tricks.
• DeepSeek-ViT was unfrozen, and they were jointly posttrained by SFT and RL, then consolidated by on-policy distillation. Same as V4.

KV cache compression

The V4.1 cuts down the KV cache further than V4, by changing the attention mechanism architecture itself. They called their innovation the "Causal Encoder-Decoder" (CED) and the "Compressed Sparse Attention 2" (CSA2). They claimed to take inspiration from the "you only cache once decoder-decoder".

The idea is as follows. The ${\displaystyle L}$ layers of the full neural network are split to several blocks, with ${\displaystyle L_{1},\dots ,L_{b}}$ layers respectively. Each block uses causal sliding-window self-attention. Furthermore, except the first block, all other blocks have a limited amount of global cross-attention over whatever the previous block's output is. It goes as follows:

• Given an input sequence ${\displaystyle x_{1}^{(0)},x_{2}^{(0)},\dots ,x_{N}^{(0)}}$, block 1 is a transformer that produces an output sequence ${\displaystyle x_{1}^{(L_{1})},x_{2}^{(L_{1})},\dots ,x_{N}^{(L_{1})}}$, using standard causal sliding-window self-attention.
• Block 2 is a transformer that pays attention to two KV caches, a global one and a local one. The local KV cache is the same causal sliding-window self-attention, computed at every layer. The global KV cache is a cross-attention mechanism upon ${\displaystyle x_{1}^{(L_{1})},x_{2}^{(L_{1})},\dots ,x_{N}^{(L_{1})}}$, computed by block 1.
• After that, block 3 does the same, with causal sliding-window self-attention KV cache combined with cross-attention global KV cache paid upon ${\displaystyle x_{1}^{(L_{1}+L_{2})},x_{2}^{(L_{1}+L_{2})},\dots ,x_{N}^{(L_{1}+L_{2})}}$, and so on.
• Finally, block ${\displaystyle b}$ does the same, and its output on the last token is the output of the entire neural network.

To prefill this architecture, each layer needs to store its last ${\displaystyle W}$ entries in its KV cache in order to perform sliding-window-self-attention on new tokens. Each block's output, except the last, needs to be stored too, in order to perform global cross-attention on new tokens. Consequently, the total KV cache requires memory ${\displaystyle O(WL+(b-1)N)}$, which is smaller than the global KV cache ${\displaystyle O(NL)}$. It is also smaller than that of V4, which use global KV cache with consecutive ${\displaystyle m}$ tokens merged, achieving memory ${\displaystyle O(NL/m)}$.

To cut down the cost of prefilling further, they used "bounded replay" on the last block. Specifically, instead of doing the normal causal sliding-window self-attention, they simply mask out any attention on any token other than the tokens ${\displaystyle N-w+1,N-w+2,\dots ,N}$. In other words, the attention mask looks like ${\displaystyle {\begin{bmatrix}0&0&0&0&0&0\\0&0&0&0&0&0\\0&0&0&0&0&0\\0&0&0&1&0&0\\0&0&0&1&1&0\\0&0&0&1&1&1\end{bmatrix}}}$ instead of ${\displaystyle {\begin{bmatrix}1&0&0&0&0&0\\1&1&0&0&0&0\\1&1&1&0&0&0\\0&1&1&1&0&0\\0&0&1&1&1&0\\0&0&0&1&1&1\end{bmatrix}}}$. They admit that this is mathematically different from actual causal sliding-window attention, but stated that empirically, it did not harm model performance.

Other things

V4.1 has an extra embedding matrix with 768 million rows. It embeds frequent n-grams (with n=2, 3, 4). Because it has so many rows, the embedding matrix is stored as a hash table to make retrieval feasible. They call this hash table the "Engram". This adds 196B parameters for storage, above and beyond the 552B parameters of the model itself.