Mamba-3: Improved Sequence Modeling using State Space Principles

Let the state rotate as it fades, and a fixed-size memory can keep a count.

Mamba-3 makes three changes to the Mamba-2 layer. Each step writes a blend of the current and the previous token, the state rotates as well as decays, and the per-token update into the state has rank four instead of one.

A 2026 paper from the authors of Mamba. All three changes add arithmetic to a decode step that spends its time waiting on memory, and the rotation takes the model from chance to 100 on the parity task.

Mamba-3 is a recurrent layer for language models, a drop-in replacement for attention that keeps a fixed-size state instead of a growing cache. It modifies Mamba-2 in three places, each motivated by a different view of the same recurrence:

1. as a discretized differential equation: a better integration rule adds a second, lagged input term (the exponential-trapezoidal update);
2. as a linear dynamical system: complex eigenvalues let the state rotate, which lets it count, and the rotation is computed as a data-dependent rotary embedding on ${\bm B}$ and ${\bm C}$;
3. as a GPU kernel: the per-token write into the state becomes a rank-$R$ matrix product (MIMO), which raises the arithmetic per byte of a decode step.

Around the layer, the block drops Mamba-2's short convolution, normalizes and biases ${\bm B}$ and ${\bm C}$, and alternates with a separate MLP. The last section reads the results, including where the paper's own claims run ahead of its tables.

Why a recurrent model's decode step is memory-bound

A Transformer answers a question about token 4,000 by attending to token 4,000: every key and value it has computed stays in a cache that grows with the sequence. A recurrent, or linear-time, layer keeps one fixed block of numbers, the state, updates it once per token, and reads each output from it, so a million-token sequence uses the same state memory as a ten-token one.

Mamba-2 is the version Mamba-3 starts from. Per head it keeps a state ${\bm h}_t$ and updates it with values projected from the current token $x_t$: a decay $\alpha_t$ and a gain $\gamma_t$ (two scalars), a 128-number vector ${\bm B}_t$ that says which direction in the state to write along, and a 128-number vector ${\bm C}_t$ that says which direction to read from:

Here $x_t$ is one channel of the token. A head has 64 channels that share $\alpha_t$, $\gamma_t$, ${\bm B}_t$ and ${\bm C}_t$, so the state of one head is a 64 × 128 matrix, 8,192 numbers. At the paper's 1.5B scale (model width 2,048, expansion 2, head dimension 64) there are 64 heads in each of 24 layers, about 25 MB of state in bf16 in total, independent of sequence length.

Generating one token loads that state, does a little arithmetic on it, and writes it back. Per head the step reads the 8,192-number state, ${\bm B}_t$, ${\bm C}_t$, the 64 channels of $x_t$ and a scalar: 17,026 bytes at two bytes per number, by the paper's count. The arithmetic is one scaling of the state, one outer product added into it, and one matrix-vector product to read it out, which the paper's Table 2 counts as $5NP - P$ operations for state size $N = 128$ and head dimension $P = 64$: 40,896. Operations per byte moved is the step's arithmetic intensity, here 2.4 (the paper rounds it to 2.5).

An H100 does about 989 trillion dense bf16 operations per second and reads about 3.35 TB/s from its memory, so a kernel needs about 295 operations per byte before arithmetic, rather than memory traffic, sets its run time. At 2.4 the decode step sits more than a hundred times below that line, and the GPU's compute units are idle for nearly all of it. Mamba-2, Gated DeltaNet and every other fixed-state layer are in the same position, because the state is small and read once per token.

So any change that adds operations without adding bytes costs almost nothing at decode time. All three of Mamba-3's changes add operations: a third term in the update for a more accurate step, a rotation for the ability to count, and a rank-four update to use the idle compute units.

How Mamba gets its recurrence from a differential equation

Linear-attention papers write equation (1) down directly. State space models derive it from a continuous-time system, a bank of linear differential equations driven by an input signal $x(t)$:

Tokens are treated as samples of that signal taken at times $\tau_t$, a distance $\Delta_t$ apart, and the system is discretized, turned into a recurrence over the samples. In Mamba, $\Delta_t$ is a positive number the model computes from each token (a softplus of a linear projection plus a per-head bias), and $A_t$ is negative, so the system decays. A large $\Delta_t$ treats the token as a long stretch of time: the state forgets more of its past and writes the token in harder.

Over one step the system has an exact solution. Multiplying through by an integrating factor carries the old state forward by one exponential and adds the new input as an integral:

Mamba freezes $A$ at the step's right endpoint, which gives the decay $\alpha_t = e^{\Delta_t A_t}$ for the first term. The second term, the state-input integral, is where discretization rules differ, because the model only sees the integrand at the two endpoints, the previous token and the current one. Zero-order hold, the classical rule, assumes the input was constant across the step and integrates exactly, giving a gain of $A_t^{-1}(e^{\Delta_t A_t}-1)$.

Mamba-1's paper says it uses zero-order hold. Its code computes the gain as $\gamma_t = \Delta_t$, a simplification Albert Gu confirmed in the repository (issue #129), and Mamba-2 inherited it. The deployed rule therefore combines an exact exponential for the decay with a rectangle for the input: the integrand is held at its right-endpoint value ${\bm B}_t x_t$ across the step of width $\Delta_t$. Mamba-3's first contribution is to place that rule in a family, an exponential decay times a quadrature rule for the integral, and to name it exponential-Euler; any other quadrature rule then gives another member of the family.

How the exponential-trapezoidal update works

The rectangle is exact only when the integrand does not change between the two tokens, and its error grows with how much it does change. The trapezoid rule from numerical analysis averages the two endpoints instead. Mamba-3 uses a weighted average with a weight $\lambda_t$ chosen per token, which turns equation (1) into a three-term recurrence:

$\lambda_t$ is a sigmoid of a per-head projection of the current token, so it lies in $(0,1)$. At $\lambda_t = 1$ the middle term vanishes and (5) is Mamba-2's update. At $\lambda_t = \tfrac12$ it is the classical trapezoid rule.

The factor $e^{\Delta_t A_t}$ inside $\beta_t$ is the integrand's own decay. Write the integrand with $A$ frozen at $A_t$:

At the right endpoint $\tau = \tau_t$ the exponential is 1, so $g(\tau_t) = {\bm B}_t x_t$. At the left endpoint it is $e^{\Delta_t A_t}$, because an input deposited one step earlier has decayed for one step. The last two terms of (5) are therefore $\Delta_t$ times a weighted average of $g$ at the two endpoints, with weights $1-\lambda_t$ and $\lambda_t$. Drag $\lambda$ in Figure 1 to both ends and watch the amber rectangle miss the teal area in opposite directions.

A worked step with one channel and ${\bm B} = 1$. Take $\Delta = 0.5$, $A = -1$ and $\lambda = 0.7$ for every token, so $\alpha = e^{-0.5} = 0.607$, $\gamma = 0.35$ and $\beta = 0.3 \times 0.5 \times 0.607 = 0.091$. Feed the inputs $x = (1, 0, 0)$:

• Mamba-2 ($\gamma = \Delta = 0.5$): the state reads 0.500, 0.303, 0.184.
• Mamba-3: first $0.35$, then $0.607 \times 0.35 + 0.091 \times 1 = 0.303$, then 0.184.

From the second step on, the two states are identical. With constant $\Delta$ and $\lambda$, the trapezoid term writes a fraction $\lambda$ of each token immediately and the remaining $1-\lambda$ one step later, already decayed, so the total weight a past token carries is unchanged. The difference appears when the settings vary by token, because the delayed share of token $t-1$ is set by token $t$'s own $\lambda_t$ and $\Delta_t$. If the second token above had $\lambda = 0.2$ instead, its $\beta$ would be $0.8 \times 0.5 \times 0.607 = 0.243$ and the state after it 0.455, so the second token's $\lambda$ sets how much more of the first token the state carries.

The accuracy argument comes with a condition the trained model does not meet. The paper's Remark 3 says the state-input integral is discretized to second order, with local error $O(\Delta_t^3)$, provided $\lambda_t$ stays within $O(\Delta_t)$ of $\tfrac12$. The released model leaves $\lambda_t$ as an unconstrained sigmoid, and the paper's ablation (Table 8, 440M parameters) prefers that: learned $\lambda_t$ reaches 15.72 test perplexity, fixed $\tfrac12$ 15.76, and Mamba-2's $\lambda = 1$ 15.81. So Mamba-3 as trained is not a second-order method.

Order of accuracy also describes the limit $\Delta \to 0$, and $\Delta_t$ in Mamba is a learned, unitless number that is never driven to zero. The 0.09-perplexity gain from the third term is better read as a new data-dependent term in the recurrence than as reduced truncation error, and the next section shows what that term computes.

The trapezoid term as a two-tap filter

Call ${\bm v}_t = {\bm B}_t x_t$ the state-input, the vector a token contributes before any weight is applied. Then (5) reads:

The bracket is a causal filter of width two over the state-inputs, with weights chosen per token. Recurrent layers already carry a short filter of this kind. Mamba-1 runs a depthwise causal convolution of width 4 over $x$, and Mamba-2 runs one over $x$, ${\bm B}$ and ${\bm C}$ together, followed by a SiLU. Gated DeltaNet, a linear-attention model that first erases part of its state along the written key before adding the new value (the "delta" rule), and the strongest baseline in this paper, has one too: its own ablation moves average perplexity from 27.35 to 28.95 at 400M parameters when the convolution is removed.

The two filters sit at different points. The short convolution is a separate layer with its own weights, mixing raw token features before ${\bm B}$ is applied. Mamba-3's filter has no weights of its own: its taps $\beta_t$ and $\gamma_t$ come from the same $\Delta_t$, $A_t$ and $\lambda_t$ that produce the decay, and it mixes after ${\bm B}$ has been applied. Switch Figure 2 between the two layers and watch the fan-in move from the top lane to the bottom one.

A filter inside the recurrence would be a bad trade if it slowed training. Mamba-2 trains by writing the full sequence as one matrix product, ${\bm Y} = ({\bm L}\odot{\bm C}{\bm B}^{\top}){\bm X}$, where ${\bm L}$ is a lower-triangular mask: entry $(t,s)$ is the weight with which token $s$'s write reaches token $t$'s read, the product of the decays in between times the gain. That form runs on tensor cores instead of as a sequential scan. Mamba-3's mask is the same decay mask times a matrix with one extra band:

The left factor is Mamba-2's decay mask. The right factor is the width-two filter as a matrix, $\gamma$ on the diagonal and $\beta$ one below; Mamba-2 has $\operatorname{Diag}(\gamma)$ there instead. Multiplying out, the $\alpha_{s+1}$ inside $\beta_{s+1}$ completes the decay product, and every entry below the diagonal becomes

against Mamba-2's $\big(\prod\alpha_k\big)\Delta_s$. Figure 3 shows the three matrices for six tokens. Set the step size to constant and the off-diagonal entries match Mamba-2's at every $\lambda$, as in the worked example; only the diagonal shrinks. Switch the step size to vary by token and each column blends its own token's step with the next one's.

Applying the extra band costs little, and the paper's Mamba-3 training kernels run about level with Mamba-2's. With a filter inside the recurrence, Mamba-3 ships without the external convolution. The evidence is Table 5(a), at 440M parameters: a Mamba-3 with neither the trapezoid term nor the learnable ${\bm B}$/${\bm C}$ biases (a block change covered below) reaches 16.68 perplexity; adding the trapezoid gives 16.49; adding the biases gives 15.72; adding the short convolution back on top gives 15.85. That shows the convolution adds nothing once the other two are present. The table has no run with the convolution and without them, so it does not show that the new pieces do the convolution's job, and a 0.13-perplexity difference from one run each (the paper reports no seeds or variance) does not show that the convolution hurts.

Every row of that table already has the data-dependent rotation, so the paper never measures the rotation's effect on language perplexity. Its evidence is the formal-language table, covered in the next section.

The decode-time saving is smaller than removing a layer suggests. A width-4 convolution caches a three-token buffer between steps; Mamba-3's third term caches the previous ${\bm B}_{t-1}$ and $x_{t-1}$, a comparable amount. What goes is a layer, its activation and a kernel launch.

Why a rotating state can count: parity and modular counting

Parity is the smallest hard sequence task: read a string of zeros and ones and say whether the number of ones is even. A one-layer LSTM solves it, because its state passes through a tanh and multiplicative gates every step, and nonlinear updates are not constrained by the argument below. Mamba-2 scores 0.90 on it, which on the paper's scale is chance. The argument concerns layers whose update is linear in the state, which every model here is, because linearity is what allows the parallel training form.

Mamba-2's transition $\alpha_t = e^{\Delta_t A_t}$, with $\Delta_t > 0$ and $A_t < 0$, is a real number strictly between 0 and 1 multiplying every entry of the state. Between writes the state can only shrink toward the origin along the line it is on; it can never flip sign. Grazzi et al. (2025) proved the consequence: a linear recurrent layer can compute parity only if some transition has an eigenvalue outside the non-negative reals.

The paper's example fix is a rotation, ${\bm h}_t = R(\pi x_t)\,{\bm h}_{t-1}$, where $R(\theta)$ turns a two-dimensional state by angle $\theta$. Each 1 turns the state half a circle and each 0 leaves it, so after an odd number of ones it points the opposite way, and a fixed linear readout of the first coordinate changes sign. In Figure 4, set the angle to $\pi$ and press Play: the readout bar flips on every 1.

The paper attributes the limitation to real eigenvalues, which it says "cannot represent rotational hidden state dynamics", and gives $R(\pi x_t)$ as the cure. But $R(\pi) = -I$, a real matrix with both eigenvalues equal to $-1$. Parity needs a negative eigenvalue, which is the fix Grazzi et al. proposed, and Gated DeltaNet with its eigenvalue range widened to $[-1,1]$ scores 100 on parity in Mamba-3's own results table, matching Mamba-3.

A non-real eigenvalue is needed one task later. The only real numbers that return to 1 after finitely many multiplications are $+1$ and $-1$, so a real transition can cycle with period at most two. Counting modulo 5 needs a transition that returns to the identity after five steps and not before, a rotation by $2\pi/5$. Grazzi et al.'s second theorem makes this precise: counting modulo $m$, for $m$ not a power of two, needs an eigenvalue with a nonzero imaginary part. Set the angle in Figure 4 to $2\pi/5$ and the orbit closes after five ones; move it off that value and it no longer closes at five.

A complex eigenvalue factors into a decay and a turn:

In real coordinates the turn is a 2 × 2 rotation acting on a pair of state entries. The paper's Proposition 2 states the equivalence: a complex state of $N/2$ channels is a real state of $N$ numbers whose transition is one scalar decay times a block diagonal of 2 × 2 rotations, one angle per pair. A complex number is two real numbers, so this costs no extra memory. In the state-size experiments below, Mamba-3 matches Mamba-2 at half the state.

Complex numbers are not new to the Mamba line. The related-work section says they "were phased out in Mamba-1 and successors", but Mamba-1's own ablation reports a complex selective transition at 9.16 perplexity against 8.71 for the real one on language, and Mamba-1 switched to complex for its audio experiment. Mamba-3's addition is a data-dependent imaginary part that never needs complex arithmetic to evaluate.

Mamba-3 is also more restricted than the classical complex SSMs. S4D, S5 and the LRU give each channel its own complex eigenvalue, so each channel has its own decay rate and rotation speed. Mamba-3 gives each pair its own angle $\theta_t[i]$ but makes all pairs share one decay $e^{\Delta_t A_t}$, because only a shared scalar decay keeps the mask in (7) a matrix of scalars, which is what keeps training a matrix multiply.

That structure sets a ceiling. The rotations act in fixed coordinate planes, so they commute, and the transitions one layer can compose form an abelian group: the layer can hold any number of independent counters that wrap around, and it cannot track a process where the order of operations changes the result. Shuffling a deck is the standard example, because two riffle shuffles applied in the other order give a different deck. In group terms, cyclic counting is reachable and the word problem for the permutation group $S_5$ is not, at any depth, assuming the standard conjecture $\mathrm{TC}^0 \ne \mathrm{NC}^1$. The circuit class matching what Mamba-3 adds is $\mathrm{ACC}^0$, which Barrington and Thérien tied to solvable groups; the paper labels it $\mathrm{TC}^0$, a larger class that contains it.

Computing the rotation as a data-dependent RoPE

With the rotation, the recurrence to compute is Mamba-2's update with ${\bm R}_t$ applied to the previous state:

Computed as written, this loses what Mamba-2 gained. Applying ${\bm R}_t$ to the state every step is work proportional to the 8,192-number state, and a matrix-valued transition breaks the scalar mask that lets training run as a matrix multiply.

The paper moves the rotation off the state with a change of variables. Define the running rotation ${\bm P}_t = {\bm R}_0{\bm R}_1\cdots{\bm R}_t$ and the rotated state $\tilde{{\bm h}}_t = {\bm P}_t^{\top}{\bm h}_t$. Substitute ${\bm h}_t = {\bm P}_t\tilde{{\bm h}}_t$ into (9):

Since ${\bm R}_t{\bm P}_{t-1} = {\bm P}_t$ (the rotations commute), the rotation on the previous state is exactly ${\bm P}_t$, and multiplying both sides by ${\bm P}_t^{\top}$ removes it. The transition is a scalar again, and the leftover rotation lands on the projections:

Instead of turning the state, the layer turns ${\bm B}$ and ${\bm C}$, each by the rotation accumulated up to its own position. Follow what the output at position $t$ reads from a write at position $s$. The weight is a dot product of two rotated vectors, and since ${\bm P}_t = {\bm P}_s{\bm R}_{s+1}\cdots{\bm R}_t$ with orthogonal factors, the shared prefix ${\bm P}_s$ cancels:

The pair is turned by the sum of the per-step angles of tokens $s+1$ through $t$, and by nothing else. The same cancellation makes rotary position embeddings (RoPE) encode relative distance, which is why the paper calls this a data-dependent RoPE. In Figure 5, move the key position and watch which bars add up on the dial.

Ordinary rotary embeddings fix a frequency per dimension and multiply it by the absolute position, so the angle between two tokens depends only on their separation. Mamba-3 keeps the mechanism and changes where the angle comes from: each token projects its own, and the layer carries the running sum. With $\Delta_t = 1$ and a constant $\theta$, ordinary RoPE comes back. In the released kernels the per-step angle is $\Delta_t \cdot \pi\tanh(\cdot)$, at most half a turn times that token's step size, and the accumulated angle is kept modulo $2\pi$. At decode it is 32 numbers per head, updated with one add and one sine and cosine per token. The scan kernel sees a scalar decay and real vectors, as it did in Mamba-2, and no complex arithmetic runs anywhere.

In the released code only half the state rotates: the default rope_fraction is 0.5, so of the 128 state dimensions the first 64 carry rotations (32 angles) and the other 64 behave as a plain real decaying state, which a readout can reach regardless of what lies between write and read. The paper does not vary this fraction, so how much of the state needs to rotate is unmeasured. Proposition 2 also stacks the real state as all real parts followed by all imaginary parts, while writing the rotation as a block diagonal of adjacent 2 × 2 blocks; those need different orderings. The kernel for the ordinary rank-one layer (SISO, named in the next section) interleaves adjacent pairs, and the TileLang MIMO kernel pairs dimension $i$ with $i + N/2$; the math is consistent, but the proposition cannot be copied into code as printed.

Combining the rotation with the three-term update gives the recurrence Mamba-3 runs. Each state-input keeps the rotation accumulated up to its own position, so the lagged term carries ${\bm P}_{t-1}$, not ${\bm P}_t$:

The paper writes (10) and (11) with a plain ${\bm h}_t$, the same symbol as the unrotated state of (9), while its appendix proofs call it $\tilde{{\bm h}}_t$. The outputs $y_t$ agree between the two forms, which is all the propositions claim; the states differ by the accumulated rotation, which matters when inspecting a cached state.

Below is one decode step for one head at the paper's shapes. Line 2 lists what is carried between tokens: the state, the running angles, and the previous token's ${\bm B}$ and $x$. Two lines have no counterpart in the main-text equations. $D$ is a per-head skip that adds a scaled copy of the input to the output, present in every Mamba implementation since Mamba-1. The final $\mathrm{silu}(z)$ is the block's output gate, which the paper writes only in Appendix C.

# one Mamba-3 decode step, one head.  d_state = 128, headdim = 64
# carried between tokens: h (64,128), phi (32,), B_prev (128,), x_prev (64,)
z, x, B, C, dt_r, A_r, lam_r, th_r = split(in_proj(u))   # u: (d_model,)

dt    = softplus(dt_r + dt_bias)     # this token's step size, > 0
A     = -heavy_tail(A_r)             # this token's decay rate, < 0
alpha = exp(dt * A)                  # the decay itself, in (0,1)
lam   = sigmoid(lam_r)               # the trapezoid blend lambda, in (0,1)
beta  = alpha * dt * (1 - lam)       # weight on the PREVIOUS state-input
gamma = lam * dt                     # weight on the current one

B = rms_norm(B) + B_bias             # 128 numbers; C is treated the same way
C = rms_norm(C) + C_bias
phi = (phi + dt * pi * tanh(th_r)) % (2 * pi)     # 32 running angles
B, C = rotate(B, phi), rotate(C, phi)             # first 64 dims of each

h = alpha * h + beta * outer(x_prev, B_prev) + gamma * outer(x, B)
y = h @ C + D * x                    # (64,)
B_prev, x_prev = B, x                # the extra cache the third term needs
return out_proj(y * silu(z))

MIMO: a rank-4 state update for the same state size

Back to the 2.4 operations per byte. Within one head the 64 channels share $\alpha_t$, $\Delta_t$ and ${\bm B}_t$, so stacking them makes the state a 64 × 128 matrix, and each step adds an outer product of the token's 64-number value vector ${\bm x}_t$ with its 128-number ${\bm B}_t$. That outer product has rank one: 8,192 multiplications to produce 8,192 numbers of update.

Widen both factors: let ${\bm x}_t$ be $P \times R$ and ${\bm B}_t$ be $N \times R$, and the update becomes a matrix product of rank $R$. The result has the same 64 × 128 shape, so the state does not grow, and the extra bytes are small because $R$ is small next to $N$ and $P$. The operation count grows roughly $R$-fold. In signal-processing terms this is the move from a single-input single-output (SISO) system to a multi-input multi-output (MIMO) one, hence the name. A rank-$R$ update is the more accurate description, because the $R$ "inputs" are $R$ learned rescalings of one projection, not $R$ separate data channels; the paper describes MIMO as motivated by inference efficiency.

The paper's introduction says MIMO "increases decoding FLOPs by up to $4\times$". Its Table 2 formulas give a SISO step $5NP - P$ operations and a rank-$R$ step $4NPR + NP - PR$. At the trained configuration, $N = 128$, $P = 64$, $R = 4$, that is 40,896 against 139,008, a ratio of 3.40. For large $N$ and $P$ the ratio tends to $(4R+1)/5$, also 3.4 at $R = 4$, because the $NP$ operations that scale the old state do not grow with $R$. A true $4\times$ needs $R \ge 5$. The bytes moved go from 17,026 to 18,946, up 11%.

Table 2's byte count also omits two things. Summing the tensor shapes it lists gives one $P$-sized tensor more than the printed formula, and neither counts writing the updated state back. Counting the full round trip, the intensity is about 1.2 for SISO and 3.9 at $R = 4$, so the step is even more memory-bound than the paper says. On the paper's count the distance to the 295 break-even shrinks from about 120× to about 40×; on the full round trip, from about 240× to about 75×.

The rank also costs training time. A rank-$R$ recurrence is $R$ state updates sharing one decay, summed into one state, and $R$ readouts, each contracting against that sum:

Every write reaches every read, so an algorithm that materializes all the write-read pairs runs the SISO computation $R^2$ times, 16 times at $R = 4$. The chunked algorithm avoids most of that. SSD, the structured state space duality algorithm Mamba-2 trains with, splits the sequence into chunks, does a quadratic computation inside each chunk and a recurrence between chunks; the quadratic part is what grows as $R^2$. Shrinking the chunk by a factor of $R$ holds that part fixed while the number of chunks grows by $R$, for an $R$-fold cost overall. The released code uses chunk size 64 for SISO and $64/R$ for MIMO. Measured, the MIMO training kernels are about 2× slower at $R = 4$, better than 4× because the extra compute overlaps with memory movement.

The rank also adds parameters. ${\bm B}$ and ${\bm C}$ are shared across all heads in a layer, so widening their projections from $DN$ to $DNR$ (for model width $D$) is affordable. ${\bm x}$, ${\bm y}$ and the gate ${\bm z}$ are per head, and widening their projection from $DP$ to $DPR$ would multiply the largest part of the parameter count. Mamba-3 instead keeps the ordinary projection and rescales its output into $R$ copies with a learned, data-independent vector, $DP + PR$ parameters per head. To keep the total parameter count equal to the SISO model's, Mamba-3 offsets the extra MIMO parameters by narrowing the MLP: at 1.5B its inner width drops from 4,096 to 3,824, a 6.6% cut.

Earlier MIMO state space models, S5 and the LRU, used the term for the opposite purpose: they replaced S4's bank of independent per-channel SSMs with one shared multi-channel SSM to shrink the recurrent state, and the paper describes them as trading some expressivity for simpler training. Mamba-3 keeps the state at Mamba-2's size and does more operations per step.

The Mamba-3 block, layer by layer

The changes so far are inside the recurrence. The block around it changes too; Figure 7 switches between the two blocks.

The first change is an RMSNorm on ${\bm B}$ and ${\bm C}$ right after projection, which the paper calls BCNorm. Under the duality Mamba-2 established, ${\bm C}$ plays the role of the attention query and ${\bm B}$ the key, and the released kernel is called with $Q = {\bm C}$ and $K = {\bm B}$. Normalizing them is the SSM counterpart of the QK-norm that recent Transformers use for training stability. There is no softmax in an SSD layer; unit-RMS ${\bm B}$ and ${\bm C}$ bound the entries of ${\bm C}{\bm B}^{\top}$ directly, which Wortsman et al. identified as what stabilizes large-scale training. The paper credits BCNorm with letting it drop Mamba-2's post-gate RMSNorm, but reports no run without BCNorm anywhere.

The second change is a learnable bias added to ${\bm B}$ and ${\bm C}$ after the norm (and, in the code, before the rotation), one value per head per state dimension, initialized to ones. The initialization changes what the layer computes at the start of training. After RMSNorm each of $\hat{{\bm b}}$ and $\hat{{\bm c}}$ has unit root-mean-square over its 128 entries, so their inner product has mean 0 and a spread of about $\sqrt{128} \approx 11$, set by content. Adding ones to both adds a constant:

Each sum runs over 128 roughly centred entries of unit size, so each has a spread of about 11, like the content term. The three add in quadrature to a spread of about $\sqrt{3 \times 128} \approx 20$ around a mean of 128. At initialization most of every entry of ${\bm C}^{\top}{\bm B}$, the score the decay mask $L$ multiplies, is a data-independent constant, so the layer starts as a nearly content-blind decaying average of the sequence and learns content selectivity on top. The rotation modifies this for the rotated half: for two positions far apart the 32 accumulated angles are effectively random, those 64 dimensions' share of the constant averages out, and about 64 remains. Figure 8 samples 3,000 token pairs both ways for each initialization the paper tried.

The ablations fit that reading. Initializing the biases to zero costs 0.85 perplexity (16.57 against 15.72); a uniform $[-1,1]$ initialization, where the two biases are uncorrelated and their product sums to about zero, costs 0.35 (16.07). Any positive initialization works about equally well: $U(0,1)$ gives 15.76 and ones held frozen 15.80. The paper offers its own hypothesis, that the biases "induce a convolution-like behavior", and the constant offset is this page's reading of the same numbers.

The paper cites Yu and Erichson's block-biased Mamba as motivation. Their Theorem 4 gives two separate sufficient fixes for single-layer universal approximation: block partitioning alone, or a channel-specific bias on ${\bm B}$ alone. Mamba-3's Table 10(b) finds ${\bm B}$ alone the worst of the four options, 16.68 against 16.52 for no bias, while ${\bm C}$ alone gives 15.98 and both 15.69. The bias on the read side carries the effect, and the cited theory is about the write side.

The third change moves the output norm. Mamba-2 put an RMSNorm after the gate and before the output projection, added because larger models were unstable. Mamba-3 removes it from pure models, and the released module has it off by default. Hybrids that interleave attention layers need a norm back, and the paper's Table 9 (1.5B hybrids, needle-in-a-haystack retrieval at up to 4,096 tokens, twice the 2,048 training context) varies two things at once: where the norm sits and whether it averages per head ("grouped") or across all heads of the layer. At 4,096, across the table's three needle tasks and both norm types (six comparisons), moving the norm before the gate wins five. Grouped against layer-wide shows no clean winner: across the same three tasks at both placements, the layer-wide norm is ahead in four of six, although the caption names the pre-gate grouped version best.

The fourth change is to the stack. Mamba-1 and Mamba-2 stack one block type with no separate MLP, and both papers ablated interleaving an MLP and found it slightly worse. Mamba-3 alternates its layer with a SwiGLU MLP block in the style of LLaMA. It reverses the earlier ablations, and the paper does not ablate it; it says only that the baselines follow their own papers' recipes.

The remaining nonlinearities in the layer are a softplus on $\Delta$, a sigmoid on $\lambda$, the two RMSNorms, a SiLU on the gate, and an activation on $A$ that the paper does not mention and the code calls heavy_tail: $1+u$ for $u \ge 0$ and $1/(1-u)$ for $u < 0$, negated and clamped at $-10^{-4}$. It approaches zero slowly for negative inputs, so a moderate projection still gives a long memory, and its docstring says it improves stability at higher learning rates. No convolution or activation acts on the $x$ path.

Mamba-3 results: perplexity, state tracking, retrieval and speed

Every model in the comparison (a Transformer, Mamba-2, Gated DeltaNet, Mamba-3 SISO and Mamba-3 MIMO at $R = 4$) is trained on 100 billion tokens of FineWeb-Edu with the Llama-3.1 tokenizer and a 2,048-token context, at four sizes: 180M, 440M, 880M and 1.5B parameters. The Mamba models use expansion 2, state size 128 and head dimension 64. Downstream quality is the average of seven zero-shot tasks.

At 1.5B the zero-shot averages are 55.4 for the Transformer, 55.7 for Mamba-2, 55.8 for Gated DeltaNet, 56.4 for Mamba-3 SISO and 57.6 for MIMO: +0.6 for SISO over the best baseline, a further +1.2 for MIMO, +1.8 in total, and +2.2 over the Transformer. The ordering repeats at 880M, 440M and 180M, which is stronger evidence than any single gap. Seven tasks give each model's average a sampling error of roughly half a point; scoring all models on the same examples cancels part of that, but with one run per model and no variance reported, 0.6 points is within noise and 1.8 is not.

The 180M row does not rank anything. OpenBookQA has four choices, so chance is 25%, and all five models score between 21.8 and 23.2. ARC-Challenge, also four choices, spans 27.3 to 28.2, and WinoGrande, a binary task, spans 51.2 to 52.9. Three of the seven columns are noise, and 100 billion tokens for 180 million parameters is about 556 tokens per parameter, 28 times the 20 that Chinchilla's compute-optimal fit recommends.

Perplexity is the cleaner measure. On FineWeb-Edu at 1.5B, Mamba-2 reaches 10.47, Mamba-3 SISO 10.35 and MIMO 10.24. The total 0.23 is a 2.2% drop, 0.022 nats (0.032 bits) per token. In the paper's own Mamba-2 column, going from 880M to 1.5B parameters (11.35 to 10.47) is worth 0.081 nats, so by this page's interpolation Mamba-3's gain is about what a 10-20% larger Mamba-2 would give.

The state-size sweep bears directly on decode speed, since state size sets the bytes per step. Across 440M models trained to twice the Chinchilla token count at state sizes 16, 32, 64 and 128, Mamba-3 with half the state reaches better pretraining perplexity than Mamba-2 while moving half the state bytes per generated token.

State tracking gives the largest differences in the paper. Table 5(b) reports scaled accuracy, where 0 is chance and 100 is perfect, on parity, modular arithmetic without brackets, and modular arithmetic with brackets:

• Mamba-3: 100.00, 98.51, 87.75.
• Mamba-3 with ordinary RoPE: 1.56, 20.70, 2.62.
• Mamba-3 without the rotation: 2.27, 1.49, 0.72.
• Mamba-2: 0.90, 47.81, 0.88.
• Gated DeltaNet with eigenvalues in $[-1,1]$: 100.00, 99.25, 93.50.

The ordinary-RoPE row keeps the rotation but fixes its angles by position, and it fails too, so the angle has to depend on the token. Two points qualify the headline. Mamba-2's 47.81 in the middle column is far from chance (about 2.9× chance in raw terms), so the introduction's "no better than random guessing" is wrong for that cell. And Gated DeltaNet with negative eigenvalues, an existing linear model, ties on parity and beats Mamba-3 on both arithmetic columns; the paper calls Mamba-3 competitive, "matching parity and approaching" it.

Retrieval goes the other way, as the paper says. Extracting fields from semi-structured documents, Mamba-3 SISO trails the Transformer at 1.5B, 28.5 against 48.9 on SWDE and 23.4 against 58.4 on FDA, the expected cost of a fixed-size state. On question answering the gap is a few points: 40.1 against 46.6 on SQuAD and 64.5 against 67.5 on TriviaQA. On the easiest needle-in-a-haystack task at 4,096 tokens, twice the training length, Mamba-3 scores 88.2 against Mamba-2's 62.0. The Transformer scores 100.0 at 2,048 and 0.0 at 4,096, which the paper attributes to its RoPE failing to extrapolate past the training length.

MIMO wins perplexity and the zero-shot average at all four sizes (perplexity 16.46, 12.72, 11.11 and 10.24 against SISO's 16.59, 12.87, 11.23 and 10.35), and it loses the harder needle tasks at 4,096 tokens, 40.4 against SISO's 50.6 and 25.6 against 34.2. Its prefill takes about 20% longer, 19.44 s against 16.22 s at 16,384 tokens.

Measured latency confirms the arithmetic-intensity argument. One decode step at batch 128 on an H100, bf16, state size 128, 1.5B models, takes 0.156 ms for Mamba-3 SISO, 0.179 ms for MIMO, 0.203 ms for Mamba-2 and 0.257 ms for Gated DeltaNet. The rank-4 update does 3.4 times the operations, costs 15% more time than SISO, and is still faster than Mamba-2. Section 4.4 says SISO is fastest "across all configurations", but at fp32 with state size 64 Mamba-2's 0.295 ms beats SISO's 0.310; the table caption's narrower claim, bf16 at state size 128, holds.

End to end at 16,384 tokens, prefill plus decode, the recurrent models finish in 141 to 152 seconds and a Transformer served with vLLM (the PagedAttention engine) takes 977. The Transformer is Llama-3.2-1B, smaller than the 1.5B recurrent models, and its 977 seconds were measured at batch 16 and multiplied by 8 because batch 128 did not fit in GPU memory. Neither caveat changes the trend, only the ratio: the Transformer's cost grows with context length and the recurrent models' does not.

Mathematically the three changes are unrelated. All three meet the same two constraints: training must stay a matrix multiply, and the decode step, which is waiting on memory, must not move many more bytes. The trapezoid adds one band to a mask that already existed, the rotation adds 32 running angles per head and leaves the transition a scalar, and the rank-4 update uses compute that was idle. At 1.5B the MIMO model is 1.8 points above the best baseline on zero-shot average, and its decode step takes 12% less time than Mamba-2's (0.179 ms against 0.203 ms).