Mamba-3: Improved Sequence Modeling using State Space Principles

Let the state rotate as it fades, and a fixed-size memory can keep a count.

Mamba-3 changes three things about Mamba-2. Each step blends two tokens instead of holding one; the state turns as well as shrinks, which lets it count; and the update packs several times the arithmetic into a state that never grows.

Ask a linear-time language model whether a bit string contains an even number of ones. Mamba-2 answers at chance, however long you train it, and the cause is a restriction on a single number inside its recurrence.

A fixed state, and a GPU that waits

A Transformer answers a question about token 4,000 by looking token 4,000 up. Every key and value it has ever computed is still sitting in memory, so the cache grows with the sequence and the attention it runs over that cache grows quadratically. A recurrent, or linear-time, sequence model refuses that deal. It keeps one running summary, a fixed block of numbers called the state, updates the state once per token, and reads its answer out of the state. Nothing accumulates. A sequence of a million tokens uses the same memory as a sequence of ten.

Mamba-2 is the version of that idea Mamba-3 starts from. Per head it keeps a state ${\bm h}_t$ and updates it with three things projected from the current token: two scalars, a decay $\alpha_t$ and a gain $\gamma_t$, and a 128-number vector ${\bm B}_t$ giving the direction in the state to write along. A fourth vector, ${\bm C}_t$, gives the direction to read from:

With Mamba-3's own settings each head carries 8,192 numbers of state, a head dimension of 64 by a state size of 128, and there are 64 heads in each of 24 layers: about 25 megabytes for an entire 1.5B-parameter model, and it stays 25 megabytes no matter how long the sequence gets. The problem is what the hardware then does with those 25 megabytes.

Generating one token means loading the state out of memory, doing a little arithmetic on it, and writing it back. Count both sides. The state is 8,192 numbers per head, two bytes each; add the token's ${\bm B}_t$, ${\bm C}_t$, $x_t$ and the scalars and you move about 17 kilobytes. The arithmetic on all of that is one scaling of the state, one outer product added into it, and one contraction to read it out: about 41,000 floating-point operations. The ratio of the two, operations per byte moved, is the arithmetic intensity of the step, and here it is about 2.4.

An H100 does roughly 989 trillion bf16 operations per second and pulls about 3.35 terabytes per second out of its memory, so it needs about 295 operations per byte before arithmetic starts to be the thing that takes time. A step at 2.4 is a hundred times below that line. Practically all of the wall-clock cost of generating a token is the GPU moving the state around with its arithmetic units idle. Mamba-2, Gated DeltaNet and every other constant-memory layer sit in the same place; it is a property of a small state read once per token, not of any one design.

Read that as a budget, not a complaint. Anything you add to the layer that spends arithmetic without moving more bytes is close to invisible at decode time. Mamba-3 spends that budget three times over, and the three purchases are different in kind: a recurrence that reads two tokens per step buys a more accurate step, a state that rotates buys a capability the layer did not have, and a state update of rank four buys hardware utilisation that was going to waste.

The recurrence comes from an integral

Equation (1) looks like something you would write down directly, and every linear-attention paper does write it down directly. State space models arrive at it a longer way, and the detour is what Mamba-3 exploits. In the SSM lineage the model is a continuous system, a bank of linear differential equations driven by a signal $x(t)$:

Tokens are then treated as samples of that signal, taken at times $\tau_t$ a distance $\Delta_t$ apart, and the continuous system is discretized into a recurrence you can actually run. In Mamba, $\Delta_t$ is not a sampling rate measured in anything; it is a positive number the model projects from each token, the softplus of a linear map plus a per-head bias. A large $\Delta_t$ means the model treats this token as a long stretch of time, so it forgets more of the past and writes the current token in harder. The two jobs are not symmetric. Forgetting goes as $e^{\Delta_t A_t}$, which saturates toward zero, while the input gain is $\Delta_t$ itself, which keeps growing, so a large enough step overwrites the state.

Solving the system over one step is a standard exercise. Multiply through by an integrating factor and you get the exact answer, with the past state carried forward by one exponential and the new input arriving as an integral:

The first term is clean if you freeze $A$ at the step's right endpoint, which costs an error of order $\Delta_t^2$ per step and gives the familiar decay $\alpha_t = e^{\Delta_t A_t}$. The second term is the one every discretization scheme disagrees about, because you only ever observe the integrand at the two endpoints and have to guess what it did in between. Classical zero-order hold assumes the input was constant across the step and integrates that exactly, giving a gain of $A_t^{-1}(e^{\Delta_t A_t}-1)$.

Mamba-1 says in print that it uses zero-order hold. Its released code does not: it computes the gain as plainly $\gamma_t = \Delta_t$, and Albert Gu confirmed in the repository that this was a simplification. Mamba-2 inherited it. So the layer everyone has been running for two years uses an exact exponential for the decay and a flat rectangle for the input, a combination that had no name and no derivation. Mamba-3's first contribution is to give it one, by treating the step as an exponential factor times a quadrature rule and reading off which rule Mamba uses. The paper calls the result exponential-Euler. Once the heuristic has a slot in a family, you can pick a different member of the family.

Use both endpoints, not one

Holding the right endpoint flat across the step is the crudest quadrature rule there is. It is right when the thing being integrated does not change between the two tokens, and it is wrong in proportion to how much it does change. Numerical analysis improves on it by averaging the two endpoints instead of taking one. Mamba-3 generalises that to a weighted average with a weight the model chooses per token, which turns equation (1) into a three-term recurrence:

The $\lambda_t$ in there is the blend, a sigmoid of a per-head projection of the current token, so it lives in $(0,1)$ and every token picks its own. Set $\lambda_t = 1$ and $\beta_t$ vanishes and you are back in Mamba-2. Set it to $\tfrac12$ and you have the classical trapezoid rule applied to the input integral.

The $e^{\Delta_t A_t}$ sitting inside $\beta_t$ looks at first like a second, separate forgetting knob, and it is not one. Give the integrand a name:

At the right endpoint $\tau = \tau_t$ the exponential is 1, so $g(\tau_t) = {\bm B}_t x_t$. At the left endpoint the exponential is $e^{\Delta_t A_t}$, because a contribution deposited a whole step ago has had a whole step to decay. So $\beta_t {\bm B}_{t-1}x_{t-1}$ is nothing but $(1-\lambda_t)\Delta_t$ times the integrand at the left endpoint, and the recurrence is a plain weighted average of the two values the model can see. Drag the figure below to both ends and watch the amber box miss the teal area in opposite directions.

That accuracy argument comes with a condition the shipped model does not meet. The paper's Remark 3 says the state-input integral is discretized to second order, with per-step error $O(\Delta_t^3)$, but only if $\lambda_t$ stays within $O(\Delta_t)$ of $\tfrac12$. The shipped model does not do that: $\lambda_t$ is an unconstrained sigmoid, free to sit anywhere in $(0,1)$, and the ablation in the appendix says that is the better choice. At 440M parameters the learned blend reaches 15.72 test perplexity, pinning $\lambda_t = \tfrac12$ gives 15.76, and falling back to Mamba-2's $\lambda_t = 1$ gives 15.81. The version that is provably second order loses to the version that is not.

There is a second reason to distrust the order-of-accuracy framing here. Orders of accuracy describe what happens as $\Delta \to 0$, and $\Delta_t$ in Mamba is a learned number with no units that nobody ever shrinks. So the 0.09 perplexity improvement from the third term is not the truncation error going away. It is a genuinely new term in the recurrence, with its own data-dependent weight, and the next section is about what that term does.

A two-tap filter inside the recurrence

Group equation (5) differently. Call ${\bm v}_t = {\bm B}_t x_t$ the state-input, the vector this token contributes to the state before any weight is applied to it. Then the update reads:

The bracket is a causal filter of width two applied to the state-inputs, with weights the model chooses per token. Modern recurrent layers all contain a filter like that. Mamba-1 runs a depthwise causal convolution of width 4 over $x$; Mamba-2 runs it over $x$, ${\bm B}$ and ${\bm C}$ together, followed by a SiLU. Gated DeltaNet has one too, and calls it crucial: removing it moves that model's average perplexity from 27.35 to 28.95 at 400M parameters. Gated DeltaNet, which turns up as the strongest baseline throughout this paper, is the linear-attention cousin of these models: instead of only adding to its state each step it first erases part of what is already there, and that erase is what its "delta" refers to.

The two filters sit at different points in the pipeline. A short convolution is a separate layer with its own weights, standing in front of the recurrence and mixing tokens before the state-input is even formed. Mamba-3's filter has no weights of its own: its taps are $\beta_t$ and $\gamma_t$, which came out of the same $\Delta_t$, $A_t$ and $\lambda_t$ that produced the decay, and it acts after ${\bm B}_t$ has been applied. Switch the figure between the two and watch the fan-in move from one stage to the other.

Putting the filter inside the recurrence would be a bad trade if it cost the training algorithm. It does not. Mamba-2 trains by writing the entire sequence as one matrix, ${\bm Y} = ({\bm L}\odot{\bm C}{\bm B}^{\top}){\bm X}$, where ${\bm L}$ is a lower-triangular mask holding the accumulated decays; that form is what lets a recurrence run on tensor cores instead of as a sequential scan. Mamba-3's mask is the same object with one extra band:

The left factor is Mamba-2's decay mask unchanged. The right factor is the width-two filter written as a matrix: $\gamma$ on the diagonal, $\beta$ one step below, zeros elsewhere. Mamba-2 has the same product with $\operatorname{Diag}(\gamma)$ on the right, so the second band is the entire structural change, and the product stays cheap to apply. Training cost is unchanged and the paper's released kernels bear that out: its Mamba-3 kernels run about level with the Mamba-2 ones.

With that filter inside, the external convolution can go, and Mamba-3 ships without one. Table 5(a) is the evidence for that decision, and it supports a narrower claim than the headline. At 440M parameters, a Mamba-3 with neither the trapezoid term nor the learnable biases on ${\bm B}$ and ${\bm C}$, a block change covered below, scores 16.68 perplexity; adding the trapezoid alone gives 16.49; adding the biases too gives 15.72; putting the short convolution back on top gives 15.85. So the convolution adds nothing once the other two pieces are there. What the table does not contain is a run with the convolution and without them, which is the comparison that would show the new pieces are doing the convolution's job and not merely making it redundant. And 0.13 perplexity at a single seed, with no variance reported anywhere in the paper, is not enough to conclude the convolution actively hurts.

Notice also what the ladder does not contain: the rotation. Every row of it already carries the data-dependent angle, so nothing in the paper prices the second of the three changes on language perplexity at all. Its case rests entirely on the formal-language table, which is the next section.

The saving at decode is also smaller than "we removed a layer" suggests. A width-4 convolution has to carry a three-token buffer between steps; Mamba-3's third term has to carry the previous ${\bm B}_{t-1}$ and $x_{t-1}$, which is a comparable pile of numbers. What is genuinely gone is a layer, its activation and a kernel launch, and with them one more piece of the block that nobody could derive from the model class.

Let the state turn

Parity is the smallest hard problem in sequence modelling: read a string of zeros and ones, say whether the number of ones is even. A single-layer LSTM solves it. Mamba-2, at any size the paper tried, does not. The LSTM gets there by being nonlinear: its state passes through a tanh and a multiplicative gate every step, so no eigenvalue argument constrains it. Everything below is about layers whose per-step update is linear in the state, which is the price all of these models pay for the parallel training algorithms they run on.

Look at what Mamba-2's transition can be. $\alpha_t = e^{\Delta_t A_t}$ with $\Delta_t > 0$ and $A_t < 0$ is a positive real number strictly between 0 and 1, and it multiplies the entire state. Between writes, the state can only shrink toward the origin along the ray it is already on. It cannot cross to the other side. Grazzi et al. (2025) turned that observation into a theorem: to compute parity, some transition in the model has to have an eigenvalue that is not a non-negative real.

The paper's fix, in its own notation, is to give the state a rotation: ${\bm h}_t = R(\pi x_t)\,{\bm h}_{t-1}$, where $R$ turns a two-dimensional state by an angle. Each 1 turns it half a circle, each 0 leaves it alone, and after an odd number of ones it is pointing the opposite way, which any linear readout can see. Play the sequence below with the angle set to $\pi$ and the readout bar flips on every 1.

Here the paper's prose slips. It says the limitation "arises from restricting the eigenvalues of the transition matrix to real numbers, which cannot represent rotational hidden state dynamics", and then gives $R(\pi x_t)$ as the cure. But $R(\pi) = -I$: a real matrix, with real eigenvalues, both equal to $-1$. Parity needs a negative eigenvalue, not a complex one. That is precisely the fix Grazzi et al. proposed, and Gated DeltaNet with its eigenvalue range widened to $[-1,1]$ scores 100 on parity in Mamba-3's own results table, matching Mamba-3.

What a genuine rotation adds shows up one task later. Products of real diagonal matrices stay real, and the only real numbers of finite order are $+1$ and $-1$, so a real transition can cycle with period two and no longer. Counting modulo 5 needs a transition that returns to the identity after five steps and not before, which is a rotation by $2\pi/5$. Grazzi et al. prove exactly this: counting modulo $m$ for $m$ not a power of two needs an eigenvalue outside the reals. Slide the angle in the figure to $2\pi/5$ and the orbit closes after five ones; nudge it off that value and the closure walks away from five, so no fixed readout counts anything.

A complex eigenvalue does both of those jobs at once. Split the transition into its two factors:

The real part sets the shrinking and the imaginary part sets the turning. In real coordinates the second factor is a two-by-two rotation acting on a pair of state slots, so a complex state of $N/2$ channels is a real state of $N$ numbers whose transition is a scalar decay times a block diagonal of rotations. That doubling costs nothing, because a complex number was two real numbers all along; the doubling is bookkeeping and the memory is unchanged. The state-size result in the paper runs the other way, with Mamba-3 matching Mamba-2 at half its state.

The related-work section says complex parameterizations "were phased out in Mamba-1 and successors". Mamba-1 demoted them without dropping them: its own ablation reports a complex selective transition at 9.16 perplexity against 8.71 for the real one on language, and it switched back to complex for its audio experiment. What is new in Mamba-3 is not complex numbers in a recurrent model, it is a data-dependent imaginary part that never requires complex arithmetic to evaluate.

Mamba-3 takes a narrower version of that than the classical complex SSMs did. S4D, S5 and the LRU give every channel its own complex eigenvalue, so each channel has its own decay rate and its own rotation speed. Mamba-3 gives every channel its own angle $\theta_t[i]$ but forces all of them to share one decay $e^{\Delta_t A_t}$. The restriction is deliberate: only a shared scalar decay keeps the mask in equation (7) a scalar-weighted triangular matrix. That is what keeps training a matrix multiply.

There is also a ceiling, and the paper is quieter about it. The rotations all act in fixed coordinate planes, which means they commute with each other, which means the transitions a single layer can compose form an abelian group. In plain terms one layer can hold any number of independent counters that wrap around, and it cannot track a process where the order of operations changes the answer: shuffling a deck by repeated riffles is the standard example, because two shuffles applied in the other order land somewhere else. Cyclic counting is reachable; the word problem for a non-solvable group like $S_5$ is not, at any depth, assuming $\mathrm{TC}^0 \ne \mathrm{NC}^1$. In the standard circuit-complexity ladder the regime Mamba-3 unlocks is $\mathrm{ACC}^0$, the class Barrington and Thérien matched to solvable groups, and the paper labels it $\mathrm{TC}^0$, which contains it but is looser than the result deserves.

Carrying the rotation as one angle

Written out, the recurrence that has to be computed is Mamba-2's update with a rotation inserted in front of the state:

Running that as written would undo everything Mamba-2 gained. The state is 8,192 numbers per head, so applying ${\bm R}_t$ to it is work proportional to the entire state, and worse, a matrix-valued transition breaks the triangular mask that lets training run as a matmul. So the rotation has to go somewhere else.

Define the running rotation ${\bm P}_t = {\bm R}_0{\bm R}_1\cdots{\bm R}_t$ and change variables to $\tilde{{\bm h}}_t = {\bm P}_t^{\top}{\bm h}_t$. Substituting ${\bm h}_t = {\bm P}_t\tilde{{\bm h}}_t$ into (9) gives ${\bm P}_t\tilde{{\bm h}}_t = e^{\Delta_t A_t}{\bm R}_t{\bm P}_{t-1}\tilde{{\bm h}}_{t-1} + \Delta_t{\bm B}_t x_t$, and since ${\bm R}_t{\bm P}_{t-1} = {\bm P}_t$ the rotation on the previous state is exactly the one the change of variables already carries. Cancel ${\bm P}_t$ off the front of both sides. The transition is a bare scalar again, and what was left over lands on the projections:

So instead of turning the state you turn ${\bm B}$ and ${\bm C}$ as they go in and out, each by the rotation accumulated up to its own position. That is also why the paper calls the trick a data-dependent rotary embedding. Follow what happens when the model reads, at position $t$, something written at position $s$. The weight on that contribution is a dot product of two rotated vectors, and because ${\bm P}_t = {\bm P}_s{\bm R}_{s+1}\cdots{\bm R}_t$ and the rotations are orthogonal and commute, the two long cumulative rotations cancel down to the ones strictly in between:

So the pair is turned by the sum of the per-step angles between the two positions and by nothing else. This is the same cancellation that makes rotary position embeddings encode relative distance.

Ordinary rotary embeddings fix a frequency per dimension and multiply it by the absolute position, so the angle between two tokens is set entirely by their separation. Mamba-3 keeps the mechanism and changes the source of the angle: each token projects its own, and the model carries the running sum. Fix $\Delta_t = 1$ and hold $\theta$ constant and you get ordinary rotary embeddings back. In the released kernels the per-step angle is $\Delta_t \cdot \pi\tanh(\cdot)$, so it is at most a half turn scaled by that token's own step size, and the accumulated angle is kept modulo $2\pi$. At decode it is a vector of 32 numbers per head, one add and one sine and cosine per token. No complex arithmetic runs anywhere at inference; the scan kernel sees a scalar decay and real vectors, just as it did in Mamba-2. That is the point of the change of variables: it moves the rotation off the state and onto ${\bm B}$ and ${\bm C}$ while leaving every output unchanged.

Two details in the released code change how you should read those equations. The first is that only half the state dimensions rotate: the default rotary fraction is $0.5$, so with a state size of 128 the first 64 slots carry rotations and the other 64 run as an ordinary real decay SSM. Half the state therefore stays a plain Mamba-2 memory that a readout can reach without knowing how far back the write happened, while the other half carries position. The paper does not sweep the fraction, so how much of the state has to rotate is unmeasured. The second is that the paper writes the real state as all real parts stacked on all imaginary parts, while the rotation is written as a block diagonal of two-by-two blocks; those two conventions need different orderings, and the kernels use the interleaved one. Neither is a mathematical error, and neither survives being transcribed literally.

Combining the rotation with the three-term recurrence gives the update Mamba-3 actually runs. Each state-input keeps the rotation accumulated up to its own position, which for the lagged term means ${\bm P}_{t-1}$ rather than ${\bm P}_t$:

Written out with the shapes the paper trains at, the step fits in twenty lines. Everything carried between tokens is named on the second line, and there are four such things: the state, the running angles, and the previous token's ${\bm B}$ and $x$. Two lines have no counterpart in any equation above them. $D$ is a per-head skip that adds a scaled copy of the token straight to the readout, which the paper's equations leave out and every Mamba implementation has carried since Mamba-1. The final $\mathrm{silu}(z)$ is the block's gate, which the paper folds into its architecture figure and never writes down.

# one Mamba-3 decode step, one head.  d_state = 128, headdim = 64
# carried between tokens: h (64,128), phi (32,), B_prev (128,), x_prev (64,)
z, x, B, C, dt_r, A_r, lam_r, th_r = split(in_proj(u))   # u: (d_model,)

dt    = softplus(dt_r + dt_bias)     # this token's step size, > 0
A     = -heavy_tail(A_r)             # this token's decay rate, < 0
alpha = exp(dt * A)                  # the decay itself, in (0,1)
lam   = sigmoid(lam_r)               # the trapezoid blend lambda, in (0,1)
beta  = alpha * dt * (1 - lam)       # weight on the PREVIOUS state-input
gamma = lam * dt                     # weight on the current one

B = rms_norm(B) + B_bias             # 128 numbers; C is treated the same way
C = rms_norm(C) + C_bias
phi = (phi + dt * pi * tanh(th_r)) % (2 * pi)     # 32 running angles
B, C = rotate(B, phi), rotate(C, phi)             # first 64 dims of each

h = alpha * h + beta * outer(x_prev, B_prev) + gamma * outer(x, B)
y = h @ C + D * x                    # (64,)
B_prev, x_prev = B, x                # the extra cache the third term needs
return out_proj(y * silu(z))

This page writes the rotated state as $\tilde{{\bm h}}_t$ throughout; the paper writes both of those recurrences with a plain ${\bm h}_t$, the same name as the state in the un-rotated system. They cannot be the same object: one recurrence applies ${\bm R}_t$ to the previous state and the other does not. The proofs in the appendix name it $\tilde{{\bm h}}_t$ and are correct. The outputs $y_t$ agree between the two forms, and that is all the propositions claim. The states differ by the accumulated rotation, which matters the moment you go looking at a cached state.

More arithmetic, same state

Back to the 2.4 operations per byte. Inside one head, Mamba-2 runs the same recurrence 64 times over, once per dimension of the head, with all 64 copies sharing the same $\alpha_t$, $\Delta_t$ and ${\bm B}_t$. Stack the 64 copies and the state becomes a matrix of head dimension by state size, and the thing written into it each step is an outer product of the token's 64-number value vector with its 128-number ${\bm B}_t$. That product has rank one: 8,192 numbers of arithmetic to produce 8,192 numbers of state.

Widen both factors. Let ${\bm B}_t$ become $N\times R$ and ${\bm x}_t$ become $P\times R$, and the outer product becomes a matrix product of rank $R$. Its result is the same shape as before, so the state does not grow by a single number, and the memory traffic barely changes because $R$ is small next to $N$ and $P$. The arithmetic multiplies. In signal-processing terms this is the move from a single-input single-output system to a multi-input multi-output one, which supplies the name, though a rank-$R$ factorisation of the state update is the more accurate description. The paper's own MIMO instantiation paragraph says why: the $R$ "inputs" are $R$ learned rescalings of one projected scalar, not $R$ data channels, and its related-work section adds that the formulation is motivated by computation and not by state space theory.

The paper's own headline for this is that MIMO "increases decoding FLOPs by up to $4\times$". Its Table 2 formulas say otherwise. A SISO step costs $5NP - P$ operations and a rank-$R$ step costs $4NPR + NP - PR$; at the configuration Mamba-3 actually trains, $N = 128$, $P = 64$, $R = 4$, that is 40,896 against 139,008, a ratio of 3.40. The limit for large $N$ and $P$ is $(4R+1)/5$, which is also 3.4 at $R = 4$; the $+NP$ term for scaling the old state does not grow with $R$ and dilutes the naive intuition by a fifth, so a genuine $4\times$ needs $R \ge 5$. Against that 3.4 times the arithmetic, the bytes moved go from 17,026 to 18,946, up 11 per cent.

The byte count in that table deserves a look too. Summing the tensor shapes it lists comes to one $P$-sized tensor more than the printed denominator, and neither version counts writing the updated state back to memory. Count the full round trip and the intensity falls to about 1.2 for SISO and 3.9 at $R = 4$. That makes the step more memory-bound rather than less, so the argument gets stronger: there is even more idle arithmetic to fill than the paper claims. On the paper's own count the gap to the hardware's break-even closes from about 120 times to about 40; on the full round trip, from about 240 to about 75. Either way the step stays well inside the memory-bound region.

The rank gets paid for in training. A rank-$R$ MIMO recurrence is $R$ state updates sharing one decay, summed into a single state, and $R$ readouts each contracting against that sum. Every write has to reach every read, so a training algorithm that materialises those pairings runs the ordinary SISO computation $R^2$ times.

Run that as written and training costs $R^2$ times as much, which at $R=4$ would be fatal. The chunked algorithm rescues it. SSD, the structured state space duality algorithm Mamba-2 trains with, splits the sequence into chunks, does a quadratic computation inside each chunk and a recurrence between chunks, and the quadratic part is the piece that grows as $R^2$. Shrink the chunk by a factor of $R$ and that term is held fixed while the number of chunks grows by $R$, leaving an $R$-fold increase overall. The released code does just that: chunk size 64 for SISO, 64 divided by the rank for MIMO. Measured, the MIMO kernels run about twice as slow at $R = 4$, better than the $4\times$ the accounting allows because the extra arithmetic overlaps with memory movement.

Parameters have to be paid for too, and the accounting gets fiddly. ${\bm B}$ and ${\bm C}$ are shared across all heads in Mamba, so widening their projections by $R$ costs one extra $DNR$ block for the whole layer, which is affordable. But $x$, $y$ and the gate $z$ are per head, and widening those would multiply the dominant part of the parameter count by $R$. So Mamba-3 does not widen them: it keeps the ordinary projection and rescales its output into $R$ copies with a learned, data-independent vector, which costs $PR$ extra numbers per head instead of $(R-1)DP$. Whatever remains is paid for by narrowing the MLP, from 4,096 to 3,824 at 1.5B, a 6.6 per cent cut.

The earlier MIMO state space models used that word for the opposite purpose. S5 and the LRU replaced S4's bank of independent per-channel SSMs with one shared multi-channel SSM specifically to shrink the recurrent state, and the smaller state is what let them use a parallel scan. Mamba-3 keeps the state at its old size and spends arithmetic instead, so the same label covers two opposite motivations. The paper's gloss that the earlier work traded away expressivity is not settled either; S5 explicitly declines that claim about its own tying of the output matrices.

The block, piece by piece

The three ideas so far are all inside the recurrence. Around it sits the block, and Mamba-3 changes that too, in ways that are individually small and jointly account for a good part of the reported gain.

The first change is an RMSNorm on ${\bm B}$ and ${\bm C}$ just after they are projected, which the paper calls BCNorm. Under the duality Mamba-2 established, ${\bm C}$ plays the role of the query and ${\bm B}$ the key, and the released code says so in its own argument names, calling the kernel with $Q = {\bm C}$ and $K = {\bm B}$. So normalising them is the SSM version of the QK-norm that modern Transformers use for stability. The mechanism is not quite the Transformer one, though: there is no softmax anywhere in an SSD layer, so nothing is being kept out of saturation. What the norm does is bound $\|{\bm B}\|$ and $\|{\bm C}\|$, and therefore bound the entries of the mask ${\bm C}{\bm B}^{\top}$, which is the mechanism Wortsman et al. identified for large-scale training stability. The paper credits this norm with letting it drop Mamba-2's post-gate RMSNorm, and reports no ablation supporting that; there is no run without BCNorm anywhere in the paper.

The second change is a learnable bias added to ${\bm B}$ and ${\bm C}$ after that norm, one per head per channel, initialised to all ones. That initialisation does most of the work, and the arithmetic shows why. After RMSNorm each of $\hat{{\bm b}}$ and $\hat{{\bm c}}$ has unit root-mean-square over its 128 entries, so their inner product is a number of order $\sqrt{128} \approx 11$ that depends on content. Add ones to both and a constant appears out of nowhere:

Each sum runs over 128 roughly centred entries of unit size, so each is itself of order $\sqrt{128}$: negligible next to the constant 128, and the same size as the content term. Sampling that out, a mask entry without the bias has mean 0 and a spread of about 11; with it, a mean of about 128 and a spread of about 20. So at initialisation roughly six sevenths of every mask entry is a data-independent constant. The layer begins life as a nearly content-blind decaying average of the sequence and learns content selectivity on top of it, rather than starting from a random selection and having to find the average.

The ablations back that reading up and complicate the paper's story about where the idea came from. Initialising the biases to zero costs 0.85 perplexity, 15.72 against 16.57; initialising them uniformly on $[-1,1]$, where they can be negative, costs 0.35. Anything positive works about equally well. The motivating prior work, Yu and Erichson's block-biased Mamba, proves that a channel-specific bias on ${\bm B}$ alone restores universal approximation to a single-layer model, and it biases only ${\bm B}$. Mamba-3's own table says ${\bm B}$ alone is the worst of the four options, 16.68 against 16.52 for no bias at all, while ${\bm C}$ alone gets 15.98 and the pair gets 15.69. The bias on the read side carries the effect, and the theory the paper cites is about the write side.

Third, the norm placement. Mamba-2 put an RMSNorm after the gate and before the output projection, added because larger models were unstable. Mamba-3 removes it from pure models, and the released module has it off by default. Hybrids that interleave attention layers need something back, and what works is a different layer in a different place: before the gate, normalised per head. The paper's ablation varies two things at once, where the norm sits and how widely it averages. Moving it before the gate wins five of the six long-context cells at twice the training length. Averaging per head instead of over the whole layer is close to a wash, winning one needle-in-a-haystack variant and losing another.

A fourth change gets a single sentence in the paper and reaches further than the others. Mamba-1 and Mamba-2 stack one homogeneous block type with no MLP in between; the Mamba block was designed as a SwiGLU block with a state space path folded into it, and both papers ablated interleaving a separate MLP and found it slightly worse. Mamba-3 alternates its layer with a separate SwiGLU block, Llama style. That is a genuine change to the stack rather than to the layer, it reverses an earlier ablation, and it is itself unablated here; the paper describes only Mamba-3's layout and says of the baselines that they follow their own papers' procedures.

What is left in the token mixer after all this is a short list: a softplus on $\Delta$, a sigmoid on $\lambda$, the two RMSNorms, a SiLU on the gate, and one activation on $A$ that the paper never mentions and the code calls a heavy-tail activation. It maps a projection to a positive number that grows linearly on one side and approaches zero like $1/(1-x)$ on the other, so a moderate activation still yields a very long memory, and its docstring says it improves stability at higher learning rates. There is no convolution and no activation on the $x$ path at all.

What the three changes buy

Every model in the comparison is trained the same way: 100 billion tokens of FineWeb-Edu, the Llama-3.1 tokenizer, a 2,048-token context, and for the Mamba family an expansion factor of 2, a state size of 128 and a head dimension of 64. Four sizes, five models, seven zero-shot tasks averaged.

At 1.5B parameters the headline numbers are +0.6 for Mamba-3 SISO over Gated DeltaNet, and a further +1.2 for the MIMO variant, for +1.8 over the best baseline and +2.2 over the Transformer. The ordering repeats at 880M, 440M and 180M, and that repetition carries more evidence than any single gap does. The seven tasks between them contribute a sampling error of roughly half a point to each model's average; scoring both models on the same examples cancels much of that, but the paper reports one seed and no variance, so a 0.6-point difference is not something to lean on and a 1.8-point one is.

The 180M row should not be used to rank anything at all. OpenBookQA has four choices, so chance is 25 per cent, and all five models score between 21.8 and 23.2. ARC-Challenge, also four choices, spans 27.3 to 28.2. WinoGrande is a binary task and spans 51.2 to 52.9. Three of the seven columns are contributing noise, and the models are seeing 100 billion tokens for 180 million parameters, roughly 28 times past the point where Chinchilla's compute-optimal fit would stop.

Perplexity is the cleaner measurement, and the gap it shows is small. On FineWeb-Edu at 1.5B, Mamba-2 gets 10.47, Mamba-3 SISO 10.35, and MIMO 10.24. That total 0.23 is a 2.2 per cent relative drop, 0.022 nats per token. To put that on a scale the reader can feel, going from 880M to 1.5B parameters in the paper's own Mamba-2 column is worth 0.081 nats, so all of Mamba-3's gain is what a Mamba-2 roughly 10 to 20 per cent larger would have given you. That is a real result at fixed parameters and it is not a step change.

The state-size sweep matters more than any single accuracy number, because state size is what sets decode latency. Across 440M models trained to twice Chinchilla-optimal tokens at state sizes of 16, 32, 64 and 128, Mamba-3 at half the state matches Mamba-2 at full state on pretraining perplexity. Read from the latency side, that is the same quality out of a model that moves half as many bytes per generated token.

State tracking gives the sharpest before-and-after in the paper, and needs the most hedging. On the formal-language suite Mamba-3 scores 100 on parity, 98.51 on modular arithmetic and 87.75 on modular arithmetic with brackets. Mamba-3 with ordinary rotary embeddings substituted in scores 1.56, 20.70 and 2.62, and Mamba-3 with the rotation removed scores 2.27, 1.49 and 0.72, so the data-dependent angle is doing the work and not the extra parameters. Mamba-2 scores 0.90, 47.81 and 0.88. The scale needs explaining before those numbers mean anything. It is scaled accuracy, where 0 means chance and 100 means perfect, so Mamba-2's 47.81 in the middle column is not chance at all; it is roughly three times chance in raw terms, and describing all three as performing no better than random guessing is wrong for that one cell. And Gated DeltaNet with negative eigenvalues, a prior linear model, ties on parity and beats Mamba-3 on both arithmetic columns, 99.25 and 93.50.

Retrieval goes the other way, and the paper is upfront about it. On extraction from semi-structured pages Mamba-3 sits well behind the Transformer, 28.5 against 48.9 on SWDE and 23.4 against 58.4 on FDA, which is the fixed-size state doing what a fixed-size state does. The gap narrows on question answering, where it trails the Transformer by a few points instead of twenty, 40.1 against 46.6 on SQuAD and 64.5 against 67.5 on TriviaQA. It does beat Mamba-2 on needle-in-a-haystack past the training length, 88.2 against 62.0 at 4,096 tokens on the easiest variant. The Transformer's 0.0 at that length is not a retrieval failure; it scores 100.0 at 2,048, and the collapse at twice the training context is rotary embeddings failing to extrapolate, which the paper says as well.

The MIMO variant trades one thing for another, which the summary numbers hide. It wins perplexity and the zero-shot average at all four sizes, and it loses the harder needle tasks at 4,096 tokens, 40.4 against SISO's 50.6 and 25.6 against 34.2. It also costs about 20 per cent more prefill time. Pick SISO for long-context retrieval and MIMO for short-context quality.

Latency closes the loop the arithmetic-intensity argument opened. One decode step at batch 128 on an H100 in bf16 with a state size of 128 takes 0.156 ms for Mamba-3 SISO, 0.179 for MIMO, 0.203 for Mamba-2 and 0.257 for Gated DeltaNet. So the rank-4 update, with 3.4 times the arithmetic, costs 15 per cent more time than SISO and is still faster than Mamba-2, which is the argument of Section 3.3 landing in four measured numbers. The paper's prose overshoots slightly by saying SISO is fastest "across all configurations": in fp32 at a state size of 64, Mamba-2's 0.295 ms beats Mamba-3's 0.310. Its table caption instead claims only the bf16, state-size-128 setting, and that narrower version holds.

End to end at 16,384 tokens, prefill plus decode, the recurrent models finish in 141 to 152 seconds and a vLLM-served Transformer takes 977. Before quoting that number, know what it measures. The Transformer baseline is Llama-3.2-1B against 1.5B-parameter recurrent models, and the 977 was measured at batch 16 and multiplied by 8 because the real thing did not fit in memory. Neither caveat touches the shape of the curve, only the size of the ratio: one cost grows with context length and the other does not.

As mathematics the three changes have nothing to do with each other. What they share is a constraint: whatever you add has to survive as a matrix multiply during training, and it has to cost close to nothing during a decode step that is already spending its time waiting on memory. A second endpoint adds one band to a mask that was already there, and a rotation adds one running angle per token while leaving the transition a scalar. Raising the rank of the update spends arithmetic that was going spare anyway. If you were optimising training throughput, none of that would look like an obvious place to push; starting from the step that generates a token, all three follow.