Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Serving a language model splits into two different jobs. Run them on separate machines, and build everything around the cache between them.

Mooncake, the platform behind Kimi, puts prompt-reading (prefill) and reply-writing (decode) on separate machine pools and turns the cluster's spare CPU memory into one shared cache for the keys and values that pass between them. A scheduler then routes every request around that cache to keep latency inside its targets.

The average prompt in Kimi's traffic runs about 7,590 tokens; the average reply, 182. Serving that mix, tens of thousands of times an hour, under a promise about latency, is the problem this paper is built around.

A model-as-a-service provider like Moonshot AI, which runs the Kimi assistant, is solving one optimization problem all day: push as many requests through the GPU cluster as possible, because throughput is revenue, without breaking the latency the users were promised. That promise has two parts. The time to first token (TTFT) is how long you wait after hitting enter before anything appears. The time between tokens (TBT) is how evenly the rest of the reply streams out after that. A service-level objective (SLO) is a contract on those numbers, and this paper writes them as multiples: a $\text{TTFT}_{P90}=10\times$ target means 90% of requests must reach their first token within ten times the latency that same request would have if it had a whole GPU to itself. It is a ratio, not a millisecond count, so a longer prompt earns a proportionally longer allowance.

The system rests on one observation: answering a single request is not one workload but two. Reading the prompt and writing the reply stress a GPU in nearly opposite ways, and a plain server that runs both on the same GPUs makes them fight. Mooncake pulls the two apart, and once you do, the object that has to travel between them, the cache of keys and values, becomes the thing everything else is organized around.

A few ideas carry the paper, and they stack in order: what the two phases are and why they differ, what the KV cache is and why it dominates long-context serving, why separating the phases (and pooling the cache) helps, how the cache is reused across requests, how the scheduler routes around it, how a single long prompt is prefilled fast, and what to do when the cluster is simply overloaded. None of the pieces is difficult on its own.

Two phases, opposite needs

Every request to a decoder-only Transformer (a LLaMA-style model, say) runs in two stages. In the prefill stage the model reads the entire prompt at once: all the input tokens go through the network in one shot, producing the first output token and filling a cache of intermediate results along the way. In the decode stage the model writes the reply one token at a time, each new token feeding back in to produce the next, until it stops.

Prefill has thousands of tokens to crunch simultaneously, so the arithmetic keeps the GPU's math units busy: it is compute-bound. And its cost grows faster than linearly with prompt length. Attention compares every token to every other token, which is quadratic in length, while the feed-forward layers are only linear; add them and total prefill compute sits between the two, so the paper calls it superlinear. (It is compute-heavy except for short prompts, where there aren't enough tokens to fill the hardware.) Going from 8k to 128k tokens is 16 times the tokens but, on the illustrative shape below, close to 50 times the compute, because the quadratic attention term takes over at long context.

Decode is the mirror image. Each step produces a single new token per request, which is a tiny amount of arithmetic, but to do it the GPU must re-read the entire model's weights and the full cache from memory. It spends its time waiting on memory bandwidth, not computing: it is memory-bound. ("Memory-bound" here means starved for bytes-per-second off the memory chips, not out of storage space; those are different problems.) The standard fix is to batch many requests so one read of the weights serves them all, which is why decode throughput climbs with batch size but with sharply diminishing returns: doubling the batch buys less and less as you approach a ceiling set by memory bandwidth. Before the figure, watch two things: the prefill curve bending steeply upward, and the decode curve flattening.

There is a sharper version of why batching cannot rescue decode, and the rest of the paper leans on it. Batching amortizes the shared model weights: read them once, use them for the whole batch. But the attention step is different, because every request in the batch carries its own cache. The measure to reach for is arithmetic intensity, the amount of compute a step does per byte it drags off memory; when that number is low and fixed, the arithmetic finishes long before the bytes arrive and the chips idle. The paper states the arithmetic intensity of decode-stage attention exactly: it is the number of attention heads divided by the number of key/value heads, and nothing else. For LLaMA2-70B that is 64 query heads over 8 key/value heads, a flat 8 whatever the batch size, because adding requests grows the attention work and the bytes it reads together, so the ratio does not move. Attention stays memory-bound no matter how large the batch, which is why decode never fully escapes the memory wall.

So the two stages suit different machines. Prefill needs raw compute and long, fat matrix multiplies; decode needs memory bandwidth and lots of concurrent requests to amortize it. Run both on one pool of GPUs, as most servers do, and a long prompt's prefill lands in the middle of everyone else's decoding and freezes it: every streaming reply skips a beat while the big prefill runs. That stall is a TBT violation, and it is the interference Mooncake sets out to remove.

The KV cache moves

The "cache of intermediate results" that prefill fills has a name: the KV cache. Inside attention, each token is turned into a query, a key, and a value. To predict the next token, the model's query attends over the keys and values of every earlier token. Those keys and values depend only on the tokens already seen, so once you have computed them there is no reason to compute them again. The KV cache stores them, and it turns each decode step from "re-attend over the whole prefix" into "compute one new token and append its key and value." (This is an inference-time trick, not part of the original Transformer, which never describes caching keys and values across generation steps.)

The cache is also the thing that physically travels with a request. Prefill produces it; decode consumes it, one step at a time, growing it by one token's worth each step. When the paper says the architecture is KVCache-centric, it means the KV cache is the object that moves from the prefill machine to the decode machine, the object that can be reused across requests, and the object whose size decides how many requests a GPU can hold.

That size is easier to grasp concretely, and it is why long-context serving is hard. For the LLaMA2-70B architecture the paper uses as a stand-in, each token's cache is two vectors (a key and a value), across 80 layers, across 8 key/value heads of dimension 128, at two bytes each: $2 \times 80 \times 8 \times 128 \times 2$ bytes, about 320 KB per token. (Only 8 key/value heads, not 64, because LLaMA2-70B uses grouped-query attention, the same cache- shrinking trick multi-query attention introduced; plain multi-head would be eight times larger.) An average 7,590-token prompt therefore carries roughly 2.3 GB of cache, so an 80 GB GPU holds only a few dozen such requests at once. Memory, not compute, caps decode batch size, and every gigabyte the cache can avoid storing or recomputing is a gigabyte of extra throughput.

Split the two, pool the cache

Mooncake stops running prefill and decode on the same machines. It splits the cluster into a prefill pool and a decode pool, each tuned for its own workload, so a long prefill can never stall a decode batch: they are not on the same GPUs anymore. Splitting the two phases was not Mooncake's invention, and the paper is clear about that. Splitwise proposed phase-splitting first and Mooncake credits it as motivating the work; DistServe and TetriInfer are concurrent and, in the authors' words, corroborate the findings. What is new here sits one level up, in how the split cluster is scheduled and how the cache is stored.

The other half of the change concerns the cache. A GPU cluster has a lot of CPU memory (ordinary DRAM) and SSD sitting mostly idle next to the accelerators. Mooncake pools all of it, plus the fast RDMA network between nodes, into one disaggregated KV cache: a big, cheap, near-GPU store for keys and values that no single GPU could hold. Blocks live in CPU memory as paged chunks, moved between machines by a per-node transfer process called Messenger that uses GPUDirect RDMA, letting the network card read and write GPU memory directly without routing through the CPU. The cache stops being a per-GPU scratchpad and becomes a cluster-wide resource.

With the pieces named, a request's life is four steps, and the KV cache is present in every one. The scheduler, a component called Conductor, first picks a prefill node and a decode node for the request. Then: (1) load whatever prefix of this prompt is already cached from the pool into the prefill node; (2) prefill the rest, in chunks, producing fresh cache blocks; (3) stream those blocks, layer by layer, across to the chosen decode node; (4) decode, generating tokens in a continuous batch. Step through it:

Step 4 turns on one phrase: continuous batching. Rather than wait for a fixed batch to fill and finish, the decode scheduler revisits the batch every single step, adding a newly arrived request and dropping a finished one on the fly (the idea from Orca, which keeps the decode GPUs busy). Mooncake's baseline, vLLM, combines continuous batching with paged KV storage and is one of the fastest open servers, but it runs prefill and decode together on the same GPUs. That coupling is what the disaggregation removes.

Reuse the cache by prefix

Now the first of the two throughput levers: don't recompute a cache you already have. Many requests share a beginning. A system prompt is prepended to nearly every call; users paste the same document and ask several questions about it; a chat replays its whole history each turn. Whenever two prompts start the same way, their keys and values for that shared start are bit-for-bit identical, because attention is causal: a token's key and value depend only on the tokens before it. So the cache for a shared prefix can be computed once and reused, and only the part where the prompts diverge has to be prefilled.

To find shared prefixes fast, Mooncake splits each prompt into fixed blocks of 512 tokens and hashes them in a chain: each block's hash folds in the hash of every block before it. Two prompts then produce the same hash for a block only if their entire lead-up to that block is identical, so matching block hashes one by one, until the first mismatch, gives the exact length of the shared prefix. Version-control systems hash commit IDs the same way. The paper's own trace shows it directly:

# two real requests from Mooncake's open trace (user text stripped)
{ "timestamp": 27482, "input_length": 6955, "output_length": 52,
  "hash_ids": [46,47,48,49,50,51,52,53,54,55,56,57, 2353, 2354] }
{ "timestamp": 30535, "input_length": 6472, "output_length": 26,
  "hash_ids": [46,47,48,49,50,51,52,53,54,55,56,57, 2366] }
# first 12 hash_ids match -> a shared prefix of 12 x 512 = 6,144 tokens

Two requests, different lengths, and their first twelve hash IDs are identical before they split (one ends 2353/2354, the other 2366). Twelve blocks of 512 tokens is a 6,144-token prefix the second request reuses instead of recomputing. The reused blocks load their keys and values straight from the pool; only the diverging suffix runs through prefill. Drag the shared prefix and watch the compute the prefill node has to do collapse:

Two limits keep this from being magic. Reuse is prefix-only: change one token in the middle of a prompt and every block after it gets a different hash, so the tail is worthless. And reuse cuts prefill compute only; decode still has to stream the full cache regardless. How much does it buy in practice? On the paper's one-hour trace, the cache hit ratio climbs from 30% to about 50% as the pool grows from 1,000 to 50,000 blocks, then flattens near 51%. So roughly half of all KV is reusable here, and the authors note it can reach far higher for some workloads. (You will see the headline "90% reusable" repeated online; that figure is from a summary of a specific chat-over-documents service, not the general result. The paper's measured ceiling on its trace is about 50%, and it warns that even the flat part of the curve does not mean bigger caches are useless, since the trace is only one hour.)

There is one more wrinkle the reuse creates. Block popularity is wildly skewed: more than half the cached blocks are never reused, while a few hot blocks (that shared system prompt) get read tens of thousands of times. Left alone, every request needing a hot block would fetch it from the one node that holds it, and that node's network would choke. So Mooncake replicates hot blocks across nodes. It does this without trying to predict future usage, through a heuristic covered next: when the scheduler routes a request away from the node holding a block, it copies the block along the way.

Schedule around the cache

Reuse sets up a genuine conflict. To reuse a cached prefix you want to route a request to the node that already holds it. But that is the popular node, so it is also the busiest, and waiting behind its queue can cost more time than the reuse saves. It is subtle because both naive answers are wrong: always chase the cache and you pile everyone onto the hot node; always balance the load and you throw away reuse and recompute everything.

Conductor resolves it by estimating, for each candidate prefill node, the actual time to first token that routing there would produce, and picking the smallest. That estimate is a sum of parts the system can predict: the queue it would wait behind, the transfer if the cache has to be fetched from another node, and the prefill itself (shorter when more of the prompt is already cached there). A model fit to offline measurements turns request length and cache-hit length into a prefill-time estimate; the Transformer's regular compute pattern keeps the error small.

# Conductor picks ONE prefill instance for a request (cache-aware)
best_prefix, best_node = find_best_prefix_match(prefill_pool, hashes)
ttft_min, chosen = INF, None
for node in prefill_pool:
    prefix  = node.prefix_len          # how much of THIS prompt it holds
    t_queue = estimate_queue_time(node)
    if best_prefix / prefix < balancing_threshold:   # local cache is enough
        t = t_queue + prefill_time(prompt_len, prefix)
    else:                                            # fetch the hot cache
        t_move = transfer_time(best_node, node, best_prefix - prefix)
        t = t_queue + t_move + prefill_time(prompt_len, best_prefix)
    if t < ttft_min: ttft_min, chosen = t, node
if ttft_min > ttft_slo or decode_pool_over_slo(): reject(429)  # too busy
return chosen

Read the loop and the tension is visible. The balancing_threshold is the knob that decides, per node, whether the remote hot cache is worth fetching or the request should recompute locally; and when a request is routed to a node that then fetches the hot block, that fetch also replicates the block to a second place. If no node can meet the SLO, Conductor rejects the request immediately with an HTTP 429 rather than admit something it cannot finish on time. Drag the cluster load and watch the winner move:

This scheduling lowers the average TTFT and raises the fraction of requests that hit their SLO. Against both random routing and plain load-balancing, it wins on real traffic because it captures reuse when reuse is cheap and walks away from it when the queue makes it expensive.

Filling a long prompt fast

A long prompt raises its own problem. A 128k-token prefill is too much compute for a single 8-GPU node to finish inside a tight TTFT, so you want to spread one request across several nodes. The usual way to spread a layer's math across GPUs is tensor parallelism (the technique from Megatron-LM, which shards each matrix multiply). It works beautifully inside one node over the fast NVLink bus, but across nodes it needs two network-wide all-reduce synchronizations per layer, and the cross-node link is far slower than NVLink (Mooncake's fabric runs at 800 Gbps, roughly 100 GB/s, about four times slower than the intra-node NVLink), so those all-reduces crush efficiency.

Mooncake instead borrows pipeline parallelism from training. It groups a handful of prefill nodes into a pipeline, splits the prompt into chunks, and hands consecutive chunks to consecutive nodes, so the nodes work different parts of the same prompt at once and only talk at the chunk boundaries. It calls this chunked pipeline parallelism. The cross-node chatter drops to a trickle, and short prompts that fit one node are never chunked, so they pay nothing. (This uses the autoregressive structure of the model: a later chunk can start once it has the earlier chunks' cache. The paper calls chunked-pipeline prefill the first use of pipeline parallelism in inference, which overstates it a little, since inter-layer pipelining for inference already existed; what is genuinely new is pipelining one long prompt's prefill to cut its TTFT.)

The second prefill technique targets memory, not speed. A prefill node has to ship the cache it produces off to the decode side, and that transfer takes time. Do it the obvious way, finish the entire prefill and then store all the cache, and the store time stacks on top, lengthening how long the request occupies the node. Since a request's memory cost is its cache size times how long it sits there, that idle store time is pure waste. But prefill runs one layer at a time, so Mooncake launches each layer's cache store the moment that layer finishes, overlapping it with the compute of the layers still to come. The overlapped prefill then takes about the longer of the two, the layer compute or the cache transfer, instead of their sum: when the store is quicker than the compute it hides behind it, and only the very last layer's store spills past the end.

# prefill is layer by layer; KV load/store ride alongside compute
for layer in range(L):
    wait(load[layer])            # this layer's cached KV has arrived
    launch(load[layer + 1])      # prefetch the next layer's KV
    h = attention_and_mlp(h, layer)
    launch(store[layer])         # ship this layer's fresh KV, async
wait_all(store)                  # only the last store spills past compute

The bars below make the overlap literal. As the context gets longer, the cache per layer grows, so the naive tail grows with it. The overlapped version tracks the longer of compute and transfer, so it only turns transfer-bound once a single layer's store outlasts a layer's compute:

This overlap lets the scheduler ignore prefill VRAM almost entirely: as long as a node can hold a single request, the cache streams out as fast as it is made, so prefill scheduling only has to watch the cache distribution and the available DRAM, not the GPU memory.

When there is not enough

Everything so far assumes you get to serve every request. A fast-growing service does not: the cluster grows slower than the traffic, so at peak it is overloaded, and the only question is which requests to turn away. This is where the objective sharpens into goodput: throughput that actually adheres to the SLO, counting only requests that fully complete. A request the system admits, prefills, then cannot decode in time is worse than useless, because it burned a prefill and produced nothing countable. So the system rejects early: it kills a doomed request before spending compute on it.

A naive early rejection makes things oscillate. When a request arrives, its decode does not start until its prefill finishes, about one prefill-duration later. So if Conductor admits or rejects based on how loaded the decode pool is right now, it is acting on a stale reading. That delay turns admission into a feedback loop with lag, and the two pools start swinging in anti-phase, one saturating while the other drains, then trading places. It is the same overshoot as a shower with a long pipe: you feel cold, crank the hot tap, and by the time the hot water arrives you are scalding, so you crank cold, and oscillate. Each pool then sits idle half the time, the very waste you were trying to avoid.

Mooncake instead judges a request against the decode load it will meet when its prefill finishes, not the load right now. Predicting that precisely would mean predicting each request's output length, which the authors found too costly and too unreliable under overload. So they predict at the system level instead: assume a uniform decode time per request, project the aggregate decode load forward to the moment the incoming prefill will land, and admit or reject against that projection. The swing damps out. Toggle the two policies and play the clock forward:

Prediction does not reject the most requests; it rejects the fewest. In the overload test, plain load-based rejection turns away 4,183 requests, early rejection 3,771, and prediction-based early rejection 3,589. It admits more steadily, not more strictly: by keeping both pools busy instead of sloshing, the same cluster finishes more work.

What it buys

The gains depend heavily on how much of the workload is long-context, so read them by setting. In production, serving Kimi, the architecture lets the cluster handle about 75% more requests than the coupled baseline. On two public long-context datasets, a three-prefill, one-decode Mooncake cluster beats a four-node vLLM cluster of the same size by 20% on arXiv summarization and 40% on the L-Eval benchmark, both while meeting the SLO. On synthetic long-context data, where a single request can be huge, the improvement ranges from 50% up to 525%; that top number is a best-case simulated ceiling, not a typical figure, and it should never be quoted bare.

The most telling result is not a throughput number but an SLO one. Replaying real traffic on matched clusters (ten prefill plus ten decode nodes against twenty vLLM nodes), almost 100% of Mooncake's requests meet the time-between-tokens target, versus 57% for vLLM, whose coupled decode stalls whenever a long prompt lands in the batch. The first-token latencies are nearly identical between the two; the difference is entirely in the smoothness of the stream, the very interference disaggregation was built to kill.

Two things to keep straight about these numbers. They come from a dummy model that copies the LLaMA2-70B architecture but carries no trained weights, so they measure serving speed and latency, never answer quality. And Mooncake's advantage is largest exactly where its design is aimed, at long prompts with modest replies, the input-heavy shape of Kimi's traffic; on short, balanced requests the coupled baseline gives up much less.

Everything above reduces to one stance: treat the KV cache as the primary resource. One request is two workloads with opposite needs, so put them on different machines. The cache that passes between them is expensive to build and often shared, so pool it across the cluster's idle memory and reuse it by prefix. Every routing and rejection decision then turns on that cache and on a predicted latency, not on a raw request count. Kimi runs on exactly this, and that is why a service can keep answering hundred-thousand-token prompts while the request rate climbs faster than the hardware.