Mistral 7B

A 7B model that outscores Llama 2 13B, with two attention changes that cut its serving memory.

Mistral 7B is a Llama-style transformer. Its paper describes what makes it cheap to run: 32 query heads that share 8 key-value heads, and a 4,096-token attention window that caps the cache every generated token has to read.

This page follows one number through the design: the key-value cache of a 32,768-token sequence, which falls from 16 GiB to 4 GiB to 0.5 GiB, and what that does to the number of sequences one GPU can serve at once.

Mistral AI released Mistral 7B in October 2023, with weights under the Apache 2.0 license. On the twelve benchmarks in the paper’s Table 2 it outscores Llama 2 13B, a model with nearly twice the parameters, on eleven, and it beats Llama 1 34B on reasoning, mathematics and code generation. Mistral re-ran every baseline through its own evaluation pipeline, so the comparisons are like for like and the numbers differ from those printed in the Llama 2 paper.

The paper says nothing about the training data, the number of training tokens or the recipe. Its architecture section lists the changes relative to LLaMA, and every one of them is about the cost of running the model. The rest is inherited from the Llama family: rotary position embeddings (RoPE, which encode a token’s position by rotating its query and key vectors through position-dependent angles), a SwiGLU feed-forward block (two parallel projections, one passed through a SiLU activation, multiplied elementwise), and RMSNorm (each vector divided by its root-mean-square, then scaled by learned gains). The paper’s text does not describe those three. The reference implementation shows them: it rotates queries and keys with RoPE, computes the feed-forward block as w2(silu(w1(x)) * w3(x)), and normalizes with RMSNorm.

Generating text is limited by memory traffic, and the part of that traffic that grows is the key-value (KV) cache. The paper applies four mechanisms to it:

• grouped-query attention, which stores 8 key-value heads instead of 32 and makes the cache 4× smaller;
• sliding-window attention, which lets each layer attend only to the last 4,096 positions;
• a rolling buffer, which uses the window to cap the cache at 4,096 slots;
• chunked pre-fill, which feeds a long prompt through in window-sized pieces.

After those come the benchmark results and the paper’s conclusion, which adds inference cost to training cost as an axis of scaling.

What a decode step reads

A language model writes one token at a time. To produce token $t$ it runs a forward pass through all its layers, picks a token, appends it to the sequence, and runs another pass for token $t+1$. Each token is conditioned on the ones before it, so the passes cannot run in parallel. This is autoregressive decoding, and one pass, called a decode step, produces one token for each sequence in the batch.

Every decode step reads two things from GPU memory. The first is the weights, whose size Table 1 fixes:

dim          = 4096     # model width: length of each token's vector
n_layers     = 32       # transformer blocks
head_dim     = 128      # width of one attention head (32 x 128 = 4096)
hidden_dim   = 14336    # SwiGLU feed-forward inner width
n_heads      = 32       # query heads
n_kv_heads   = 8        # key/value heads: 4 query heads share each one
window_size  = 4096     # sliding-window attention span W
context_len  = 8192     # sequence length used in training
vocab_size   = 32000    # SentencePiece tokens

Counting the matrices gives the parameter count. Each of the 32 layers has four attention projections (queries 4096 × 4096, keys 4096 × 1024, values 4096 × 1024, output 4096 × 4096, together 41.9M parameters) and three feed-forward matrices of 4096 × 14,336 (176.2M). The input embedding and the output projection are separate 32,000 × 4096 matrices in the reference code. The total is 7,241,732,096 parameters, 13.5 GiB at 16 bits per number, and each decode step streams all of it from memory, whether the batch holds one sequence or a hundred.

The second read is the KV cache: the stored keys and values of every earlier token, built up in the next section. It grows with sequence length, and each sequence in the batch has its own. For a 200-token chat it is 100 MiB, under 1% of the weights. For a 32,000-token document under standard multi-head attention it is 15.6 GiB, more than the weights.

The number 4096 appears twice in Table 1 for unrelated reasons. dim = 4096 is the length of each token’s vector. window_size = 4096 is how far back attention reaches. Neither is the context length, context_len = 8192, the sequence length the model was trained on.

Attention and the KV cache

An attention layer maps each token’s vector to three smaller ones per head: a query $q$, a key $k$ and a value $v$, each $d_k = 128$ numbers long in Mistral. A token’s output is a weighted average of the values of the tokens it can see. The weights come from its query: a dot product with each key (large when the two vectors point the same way), scaled, then passed through a softmax, which turns a list of scores into positive weights that sum to one. With one query per row of $Q$:

The division by $\sqrt{d_k}$ (here $\sqrt{128} \approx 11.3$) comes from the original Transformer paper. Dot products of 128-dimensional vectors have a spread that grows with the dimension, and unscaled they push the softmax to put nearly all its weight on one key, where its gradients are close to zero. The reference code applies it as scale = head_dim ** -0.5.

During generation, when the model produces token 100, the query of token 100 is scored against the keys of tokens 1 through 99, and their values are averaged. The key and value of token 42 depend only on tokens 1 through 42 (the causal mask keeps later tokens out of every layer), so they are the same at step 100 as they were at step 42. The model therefore computes each token’s keys and values once, in every layer, at the step that token enters, and keeps them. That store is the KV cache. The query is not cached: it belongs to the token being generated, is used once against every stored key, and is discarded.

In Figure 1, drag the position and watch two things grow together: the row of cached entries, and the fan of reads from the current token back to every one of them.

Per token and per layer, the cache holds one key and one value for each key-value head, each $d_{\text{head}}$ numbers wide. Across the stack:

With standard multi-head attention, one key-value head per query head, $n_{\text{kv}} = 32$:

numbers per token, 512 KiB at 16 bits. A 32,768-token sequence then needs 16 GiB of cache, more than the 13.5 GiB of weights, for one sequence.

The cache also sets the time of a step. For each cached token, a decode step does little arithmetic: each of the 32 query heads takes a 128-long dot product with the key (256 floating-point operations, counting multiplies and adds) and adds in 128 weighted value numbers (another 256). That is 512 FLOPs per head and $32 \times 512 \times 32$ layers, about 0.52 million FLOPs. To do it the GPU reads that token’s cached keys and values, 512 KiB, about 0.52 million bytes: 1 FLOP per byte read. A GPU can do on the order of 150 FLOPs in the time it takes to read one byte from its memory (a typical compute-to-bandwidth ratio, not one card’s specification), so at 1 FLOP per byte its arithmetic units are busy under 1% of the step. Decoding is memory-bandwidth bound: the time of a step is set by the bytes it reads, and a smaller cache makes the step faster in proportion.

Figure 2 draws that division. The slider changes how many tokens are cached; the buttons change $n_{\text{kv}}$ ahead of the next section.

Grouped-query attention: share the keys and values

Three factors in (2) are set by the rest of the architecture: the 2 for keys and values, $d_{\text{head}}$, and $n_{\text{layers}}$. The fourth, the number of key-value heads $n_{\text{kv}}$, can be chosen separately from the number of query heads.

Standard multi-head attention (MHA) gives each of the 32 query heads its own key head and value head: $n_{\text{kv}} = 32$. Multi-query attention (MQA; Shazeer, 2019, in a paper titled Fast Transformer Decoding: One Write-Head is All You Need) keeps the 32 query heads and gives them one shared key head and one shared value head: $n_{\text{kv}} = 1$, a cache 32× smaller. The GQA paper reports that MQA “can lead to quality degradation”: 32 heads that compute different queries must all score the same keys and average the same values.

Grouped-query attention (GQA; Ainslie et al., 2023) sits between the two. Split the query heads into $G$ groups and give each group one key head and one value head. $G = 32$ is multi-head and $G = 1$ is multi-query. Mistral uses $G = 8$: 8 key-value heads, each serving 4 query heads. The cache is 4× smaller than multi-head, 128 KiB per token, and the 32,768-token sequence needs 4 GiB instead of 16.

Picture 32 readers who each look things up in a reference book. Multi-head attention gives every reader a private book; multi-query attention makes all 32 share one; grouped-query attention gives each table of four readers its own. The readers still ask 32 different questions, and only the number of books, which are what the cache stores, goes down. The analogy breaks in one place: the 32 books of multi-head attention are 32 different learned projections, so merging them to 8 does remove information that the query heads could have used.

In code, the sharing happens at read time. One attention layer in the reference implementation, for one token:

# one attention layer, decode step, shapes for one token (batch dim omitted)
q = x @ wq                    # wq: 4096 x 4096 -> 32 heads x 128
k = x @ wk                    # wk: 4096 x 1024 ->  8 heads x 128
v = x @ wv                    # wv: 4096 x 1024 ->  8 heads x 128
cache.write(k, v)             # the cache stores 8 heads, not 32
K, V = cache.read()           # (n_cached, 8, 128) each
K = repeat_interleave(K, 4)   # (n_cached, 32, 128): query heads 0-3 read head 0,
V = repeat_interleave(V, 4)   #   heads 4-7 read head 1, and so on
out = softmax(q @ K.T / sqrt(128)) @ V    # per head, as in (1)

The layer projects to 8 key heads and 8 value heads, caches those, and copies each 4 times with repeat_interleave just before the dot products, so query heads 0 to 3 read key-value head 0, heads 4 to 7 read head 1, and so on. The smaller wk and wv also save parameters: 2 × 4096 × (4096 − 1024) = 25.2M per layer, about 805M across the model. The paper credits GQA with faster inference and lower memory during decoding, “allowing for higher batch sizes hence higher throughput”.

Figure 3 shows the wiring. The 32 query heads never change; toggle between the three settings and only the row of key-value heads and the cache bar below them change.

Grouped-query attention is not multi-query attention. All 32 query heads remain; only keys and values are shared, in groups of four, for a 4× cache reduction rather than MQA’s 32×. Mistral also trained with GQA from the start. The GQA paper’s own recipe converts an existing multi-head checkpoint by “uptraining” it with 5% of the original pre-training compute, and its finding that GQA reaches “quality close to multi-head attention” is measured on those converted models. The Mistral paper runs no comparison of GQA-8 against multi-head attention, so the quality cost of the choice in this model is not measured.

Sliding-window attention: a fixed span per layer

GQA shrinks the cache by a constant factor, and the cache still grows with every token, because every token attends to every token before it. The paper’s Figure 1 states the cost of vanilla attention: operations quadratic in the sequence length, memory growing linearly with the number of tokens.

Sliding-window attention (SWA) bounds both. The paper states the rule: the hidden state at position $i$ of layer $k$ attends to the hidden states of layer $k-1$ at positions $i-W$ through $i$. Mistral sets $W = 4096$, so the token at position 10,000 attends to positions 5,904 through 10,000 and ignores the 5,904 before them. Per layer and per token, the work and the keys that must be kept are bounded by $W$ instead of by the sequence length. The idea predates Mistral; the paper cites Sparse Transformers and Longformer for it, and the “cost linear in sequence length for a fixed window” framing is Longformer’s. With changes Mistral made to FlashAttention and xFormers, the paper reports a 2× speedup over a vanilla attention baseline at a sequence length of 16K with $W = 4096$.

Why not simply truncate the input to its last 4,096 tokens, which would bound the cost the same way? Truncation deletes the start of the sequence from every layer. The window deletes it only from each layer’s direct view, and the model has 32 layers. At layer 1, position $i$ reads input positions $i-W$ to $i$. At layer 2 it reads the layer-1 states of positions $i-W$ to $i$, and the state at $i-W$ was itself computed from inputs back to $i-2W$. Each layer adds $W$ to how far back an input can influence the output. In tokens, after $k$ layers:

Take a toy window, $W = 2$, and the six tokens the fox ran past the dog at positions 0 to 5. At layer 1, dog (position 5) attends to positions 3, 4 and 5, past the dog. fox is 4 positions back, outside the window, and contributes nothing at this layer. At layer 2, dog reads the layer-1 state of position 3, which read positions 1 to 3, so fox reaches dog at layer 2, through past. The first the is 5 back and arrives at layer 3, since three layers reach $3 \times 2 = 6$ positions.

Figure 4 draws that dependency cone. Rows are layers, input at the bottom; columns are positions. At the default, $W = 2$ and six layers, the output token’s cone reaches 12 positions back.

Figure 5 uses the real window and slides the depth through Mistral’s 32 layers. One column is one window of 4,096 tokens; the dashed line two columns from the target marks 8,192, the sequence length the model was trained on.

The paper calls 131,072 a “theoretical attention span”. It is an upper bound on how far information can travel, reached only if every one of 32 layers passes information from the far edge of its window. It is not a context window. Mistral 7B was trained on sequences of 8,192 tokens, so in training the cone never reached past two windows, and the paper does not measure how much information survives longer paths.

The rolling buffer cache

If no token attends further back than $W$ positions, then a key or value older than $W$ positions will never be read again. The cache can be a fixed array of $W$ slots per layer, a rolling buffer, with the key and value of position $i$ stored at:

Until position $W$ the buffer fills in order. At position $W$ the write wraps to slot 0 and overwrites position 0, which has already left the window of every later token. From then on the cache holds $W$ entries and stops growing. With $W = 4096$: position 4,095 goes to slot 4,095, position 4,096 to slot 0, and position 10,000 to slot $10{,}000 - 2 \times 4096 = 1{,}808$.

# decode step at position i, window W, cache = W slots per layer
k_i, v_i = project(x_i)        # rotary position already applied to k_i
cache[i % W] = (k_i, v_i)      # (4): overwrite the entry from position i - W
K, V = cache[:min(i + 1, W)]   # every filled slot, in slot order
out = softmax(q_i @ K.T / sqrt(d)) @ V   # slot order does not matter

In Figure 6 the buffer has 6 slots. Step the position past 6 and watch the write wrap. By position 9 the slots hold tokens 6, 7, 8, 9, 4, 5, out of order.

Attention does not need the slots in order. The output of (1) is a weighted sum over cached keys and values, and a sum does not depend on the order of its terms. Position enters through the rotary embedding, which the reference code applies to each key before writing it to the buffer (one_file_ref.py, lines 120–126), so each key carries its position with it into whatever slot it lands in.

The paper is loose by one position about the window’s width. Its text says position $i$ attends to positions $i-W$ through $i$, which is $W+1$ positions. Its Figure 1 caption says each token attends to “at most $W$ tokens”, and a $W$-slot buffer holds positions $i-W+1$ through $i$ during decoding. The reference code contains both: its pre-fill mask admits $i-W$ through $i$, and its decode step reads the $W$-slot buffer. At $W = 4096$ the difference is one position in 4,097, and it moves the reach of (3) from 131,072 to 131,040, both “approximately 131K”.

The paper measures the saving at 32k tokens: the buffer holds 4,096 entries instead of 32,768, an 8× cut in cache memory, “without impacting the model quality” (a statement; no table accompanies it). The savings multiply with GQA. For one 32,768-token sequence the cache is 16 GiB with multi-head attention, 4 GiB with GQA-8, and 0.5 GiB with GQA-8 and the 4,096-slot buffer.

The paper ties the smaller cache to batch size. A decode step reads the weights once for the whole batch, so memory freed from one sequence’s cache can hold another sequence, and more sequences per step means more tokens per second from the same weight read. The paper points to vLLM for deployment, a server built around managing many sequences’ KV caches in GPU memory (see the PagedAttention explainer). Figure 7 does the division for one GPU. On an 80 GiB card, 66.5 GiB remain after the weights. At 8,192 tokens per sequence, multi-head caches of 4 GiB fit 16 sequences, GQA-8 caches of 1 GiB fit 66, and 0.5 GiB rolling buffers fit 133.

On a 24 GiB card at 32,768 tokens, 10.5 GiB remain after the weights. A 16 GiB multi-head cache does not fit even once, GQA-8 fits 2 sequences, and the rolling buffer fits 21.

Pre-fill and chunking

Generation is sequential, but the prompt is known in advance. Pre-fill runs all the prompt tokens through the model in parallel, in one forward pass, and writes their keys and values into the cache. The first generated token waits for this pass.

For a long prompt, a single pass needs memory that grows with the prompt: activations for every prompt token at once, and, in a straightforward attention implementation, a matrix of scores per head that is prompt length squared (about 10⁹ scores per head per layer for a 32,768-token prompt). The paper splits a long prompt into chunks “to limit memory usage”, and it chooses the chunk size equal to the window, 4,096. Each chunk computes attention over the cache and over itself, then writes its keys and values to the buffer.

Take a 10,000-token prompt. The chunks are positions 0–4,095, 4,096–8,191 and 8,192–9,999. When the third chunk runs, the buffer holds the keys and values of positions 4,096–8,191 from the second. The chunk’s first token, position 8,192, needs the 4,096 positions before it, all in the buffer. Its last token, position 9,999, needs positions from 5,903 on: 5,903–8,191 from the buffer and 8,192–9,999 from its own chunk. So the mask over the buffer is a diagonal band, and the mask over the chunk is ordinary causal: each token sees itself and earlier chunk tokens, never later ones. Whatever the prompt length, one attention call covers at most 4,096 queries against 8,192 keys per head.

Figure 8 uses the paper’s own example: $W = 4$ and three chunks of four words, “The cat sat on”, “the mat and saw”, “the dog go to”. Step to the third chunk and find the three regions of its mask.

Results

Table 2 compares Mistral 7B with Llama 2 7B, Llama 2 13B and Code-Llama 7B on twelve benchmarks. GSM8K (grade-school math word problems) is scored 8-shot with maj@8, a majority vote over 8 sampled answers, and MATH 4-shot with maj@4. MBPP uses its hand-verified subset, and TriviaQA is run without Wikipedia contexts; the paper lists those two as differences from the Llama 2 paper’s protocol. MMLU is a 57-subject multiple-choice test of knowledge and reasoning.

Figure 9 shows one benchmark at a time, or all twelve gaps to Llama 2 13B at once. The largest gaps are on math: GSM8K 52.2 against 34.3, and MATH 13.1 against 6.0, more than double. MMLU is 60.1 against 55.6.

The abstract says Mistral 7B outperforms Llama 2 13B “across all evaluated benchmarks”. Table 2 has one exception: on NaturalQuestions Mistral scores 28.8 and Llama 2 13B 29.0, a gap of 0.2 points, well within the noise of such evaluations. The paper’s category chart does not show it, because it averages NaturalQuestions with TriviaQA (69.9 against 69.6) into one “World Knowledge” bar. Knowledge is Mistral’s relative weak spot, and the paper attributes it to parameter count: storing facts takes parameters, and it has about half as many.

The paper puts the efficiency in one number, an “equivalent model size”: the size of Llama 2 that would match Mistral 7B on a task group. On reasoning, comprehension and MMLU, Mistral 7B performs like a Llama 2 more than 3× its size. On knowledge benchmarks the ratio is 1.9×.

On code, Mistral 7B scores 30.5 on HumanEval and 47.5 on MBPP, far above Llama 2 13B’s 18.9 and 35.4. Code-Llama 7B, a Llama 2 fine-tuned for code, still scores higher on both, 31.1 and 52.5. The paper’s wording is that Mistral “approaches” Code-Llama’s coding performance without sacrificing performance on non-code benchmarks, where Code-Llama falls well behind (MMLU 36.9). Against Llama 1 34B, the abstract’s claim is scoped to reasoning, mathematics and code generation.

The paper runs no ablations. There is no Mistral 7B trained with multi-head attention or without the window, so these benchmark gains cannot be assigned to GQA or SWA; the paper presents both as inference savings. It also does not say what the model was trained on.

The release included Mistral 7B Instruct, the base model fine-tuned on public instruction datasets from Hugging Face, with “no proprietary data or training tricks”. It scores 6.84 (± 0.07) on MT-Bench, which grades multi-turn chat answers out of 10, above every 7B model in the paper’s Table 3 and comparable to the 13B chat models. On Chatbot Arena, which ranks models by pairwise human votes on an Elo scale (a chess-style rating where higher means more wins), it scores 1031 against Llama 2 13B Chat’s 1012; two 13B models in the same table score higher, Vicuna 13B at 1041 and WizardLM 13B v1.2 at 1047. In a separate human evaluation on llmboxing.com, as of October 6, 2023, Mistral 7B’s answers were preferred 5020 times and Llama 2 13B’s 4143 times.

A short section on guardrails measures the cost of a safety system prompt. With Mistral’s recommended system prompt the Instruct model’s MT-Bench score drops from 6.84 to 6.58 (with Llama 2’s system prompt, 6.38), and it declines all of a set of 175 unsafe prompts. Used as a content classifier with a self-reflection prompt, it reaches 99.4% precision at 95.6% recall on the authors’ own balanced set of adversarial and standard prompts, counting acceptable prompts as positives.

Inference as a scaling axis

The conclusion is two sentences long. It opens with the claim that “language models may compress knowledge more than what was previously thought”. It then says the field has treated scaling laws as two-dimensional, relating model capability to training cost, and that the problem is three-dimensional: model capability, training cost and inference cost.

The two-dimensional reference it cites is Chinchilla (Hoffmann et al., 2022), which asks how to split a fixed training compute budget between parameters and training tokens for the lowest loss, and answers about 20 tokens per parameter. Serving cost is not part of that objective, though Chinchilla’s abstract notes its 70B model is also cheaper to run than the larger Gopher.

The paper stops at naming the third axis. The usual reading of it, which the paper does not state and which its missing token count cannot confirm, goes like this: training compute is paid once and inference compute on every generated token, so a model meant to serve many tokens can be worth training past the Chinchilla point at a smaller size. The architecture section works on the same axis at a fixed size. GQA, the window, the rolling buffer and chunked pre-fill each reduce the memory that one served token costs.

None of the four mechanisms is new: GQA is Ainslie et al.’s, local attention comes from Sparse Transformers and Longformer, and the KV cache is standard decoding practice. The paper combines them in a 7.24B-parameter model that outscores Llama 2 13B on eleven of twelve benchmarks, whose cache costs 128 KiB per token and stops growing at 4,096 tokens, released under Apache 2.0.