A Spectral Condition for Feature Learning

Size every weight matrix and every update by the most it can stretch a vector, set that to √(fan-out/fan-in), and hidden features keep learning at every width.

Yang, Simon and Bernstein derive μP, the rule for scaling initialization and learning rate with network width, from a short linear-algebra argument about the spectral norm of each weight matrix and of each gradient update.

One SGD step changes a weight matrix by an outer product of two vectors. In a layer of width n, that update can have entries about √n times smaller than the weights and still change the next layer's features by an amount as large as the features themselves.

What should stay fixed as a network widens

Take a multilayer perceptron with hidden layers of width $n$ and make it wider: 256, then 1024, then 4096. If you keep the default initialization and one learning rate, the behaviour of training changes with $n$. Either the updates blow up and the run diverges, or the learning rate has to shrink as the model grows, and then the hidden layers barely change during training. The neural tangent parametrization (NTP) is the studied example of the second case: the wide network trains like a linear model in its parameters: the output fits the data, but the hidden representations stay where the random initialization put them. That regime is called lazy training, or the kernel regime.

Feature learning is the opposite: the hidden representations change by a meaningful amount during training, at every width. The paper makes this precise. Write $h_\ell(x)$ for the vector of activations at layer $\ell$ on input $x$, a vector of length $n_\ell$, and $\Delta h_\ell(x)$ for its change after one gradient step. The requirement, which the paper calls Desideratum 1, is

$\Theta(\cdot)$ means “grows like”: bounded above and below by constant multiples as the width goes to infinity, with the constants free to depend on depth, data and the global learning rate but not on width. A vector of length $n$ whose entries are each about 1 in size has Euclidean norm about $\sqrt n$: at $n = 4096$, a norm of 64. So the requirement says that each entry of each hidden vector is of order 1, and that each entry moves by an amount of order 1 per step. Larger and the network blows up as it widens; smaller and the change vanishes, which is the lazy regime.

When every layer changes by the same relative amount at every width, the best learning rate for a narrow network is also close to the best for a wide one, so you can tune a small model and copy its settings to a large one (hyperparameter transfer). The maximal update parametrization (μP) was built for this, and the μP and μTransfer explainer covers the transfer experiments. μP was derived with the Tensor Programs framework, which tracks the distribution of every coordinate in the infinite-width limit. This paper derives the same rules from bounds on matrix norms, and states them as one condition on each weight matrix $W_\ell$ and its update $\Delta W_\ell$. Condition 1 of the paper:

$W_\ell$ maps layer $\ell-1$ (width $n_{\ell-1}$, the fan-in) to layer $\ell$ (width $n_\ell$, the fan-out), so it is an $n_\ell \times n_{\ell-1}$ matrix. The double bar with a star is the spectral norm. For a square hidden layer the target is 1; for the output layer of a wide network with one output it is $1/\sqrt{n}$. The rest of this page explains what the spectral norm measures, why gradient updates make it the right measure, how the condition turns into an initialization scale and a learning rate for each layer, and how the common alternatives violate it.

Three sizes for one matrix

A matrix $A$ with $m$ rows and $n$ columns maps vectors of length $n$ to vectors of length $m$. The spectral norm is the largest factor by which it can lengthen a vector:

It equals the largest singular value of $A$. Two other sizes appear in the paper. The Frobenius norm treats the matrix as one long vector, $\|A\|_F^2 = \sum_{ij} A_{ij}^2$, which also equals the sum of the squared singular values. The RMS entry size is the Frobenius norm divided by the square root of the number of entries, $\|A\|_F/\sqrt{mn}$. The spectral norm answers a question the other two do not: how large can the output get for an input of a given length. The argument below uses two of its properties. It is subadditive, $\|A+B\|_* \le \|A\|_* + \|B\|_*$, and submultiplicative, $\|Av\|_2 \le \|A\|_* \|v\|_2$. The second is only an upper bound; how close a particular input comes to it depends on its direction.

For a 2×2 matrix, $A$ maps the unit circle to an ellipse. The lengths of the ellipse's two semi-axes are the singular values of $A$, the spectral norm is the longer one, and the input direction that lands on it is the top right-singular vector. Call the ratio $\|Av\|_2 / (\|A\|_* \|v\|_2)$ the alignment of $v$ with $A$. It is 1 for the top direction and falls toward the smallest singular value divided by the largest as you turn away from it.

The dense matrix in the figure has singular values 1.70 and 0.64, so every input is stretched by between 0.64 and 1.70, and its alignment never drops below 0.37. The rank-one matrix $uv^\top$ behaves differently: it multiplies the component of the input along $v$ by $\|u\|\,\|v\|$ and discards everything else, so its alignment with an input is the cosine of the angle between the input and $v$. For rank one the three norms line up: $\|uv^\top\|_* = \|uv^\top\|_F = \|u\|_2\|v\|_2$. Every gradient step of a linear layer on a single example is a rank-one matrix, so the identity applies to every such update.

A gradient step is an outer product

Take a deep linear network, $h_\ell(x) = W_\ell\, h_{\ell-1}(x)$ with $h_0 = x$, and a loss $\mathcal{L}$ computed from the output. By the chain rule the gradient of the loss with respect to $W_\ell$ is the gradient with respect to the layer's output times the layer's input, transposed. An SGD step with learning rate $\eta_\ell$ is therefore

a column vector of length $n_\ell$ times a row vector of length $n_{\ell-1}$. Its right singular vector is the input $h_{\ell-1}(x)$ that produced it. Apply the update to that same input and the alignment is exactly 1:

The update is built from the input and points along it, so the submultiplicative bound holds with equality. This is Claim 2 of the paper. It holds for the input that generated the gradient. An unrelated input is almost orthogonal to $h_{\ell-1}(x)$, so the same update barely changes its output.

Put a Gaussian initial weight matrix $W$ and a rank-one update $\Delta W$ side by side, both $n \times n$, both with spectral norm 1. A Gaussian matrix spreads its size over all $n$ singular values, so its Frobenius norm is about $\sqrt n$ times its spectral norm, while the rank-one update has Frobenius norm equal to its spectral norm. At $n = 1024$: $W$ has Frobenius norm 16 and entries of typical size $1/64 \approx 0.016$; $\Delta W$ has Frobenius norm 1 and entries of typical size $1/1024 \approx 0.001$. The update's entries are 16 times smaller, and the ratio, $\sqrt n/2$, keeps growing with width. Yet applied to the layer input (entries of size 1, norm 32), $W$ produces an output with entries of size about 0.5 and $\Delta W$ an output with entries of size 1. The paper's Table 1 lists these conversions for every shape.

The paper uses this to qualify the statement that the weights of a wide network barely move during training. In entry size or relative Frobenius norm, the change does shrink with width. In spectral norm, which measures the effect on the next layer, it does not have to. The paper's Figure 2 measures exactly this on trained networks, and we return to it in the experiment section.

The spectral condition, derived

The argument goes layer by layer. Assume layer $\ell-1$ already satisfies Desideratum 1, so $\|h_{\ell-1}\|_2$ and $\|\Delta h_{\ell-1}\|_2$ are both $\Theta(\sqrt{n_{\ell-1}})$. The input $x$ starts the recursion, with $\|x\|_2 = \Theta(\sqrt{n_0})$, which holds for images (for one-hot inputs, see the limits section). One step changes both the weights and the incoming features, so

and subtracting $h_\ell = W_\ell h_{\ell-1}$ leaves three terms:

Upper bounds. Apply the two norm properties to each term. The features are at most $\|W_\ell\|_* \|h_{\ell-1}\|_2$, and the change is at most

Insert Condition 1 and the assumption on layer $\ell-1$: each term is $\sqrt{n_\ell/n_{\ell-1}} \cdot \sqrt{n_{\ell-1}} = \sqrt{n_\ell}$. So the condition keeps features and their changes from growing faster than $\sqrt{n_\ell}$. Any spectral norm larger than Condition 1 would let them grow faster, which is why the paper calls it the maximal scaling.

Lower bounds. Upper bounds alone would also be satisfied by an update of size zero. The features must actually reach the bound up to a constant, and that needs the inputs to be aligned with the matrices. For the update, (7) already gives exact alignment: the first term of (9) has size $\|\Delta W_\ell\|_* \|h_{\ell-1}\|_2 = \Theta(\sqrt{n_\ell})$. Unless the other two terms cancel it, which would take a coincidence in high dimension, $\|\Delta h_\ell\|_2 = \Theta(\sqrt{n_\ell})$, as required.

For the initial weights, the paper's Claim 1 supplies the alignment. A Gaussian matrix with entry standard deviation $\sigma$ has spectral norm close to $\sigma(\sqrt{n_{\ell-1}} + \sqrt{n_\ell})$, a standard result of random matrix theory, while on any fixed input it gives $\|Wh\|_2 \approx \sigma \sqrt{n_\ell}\, \|h\|_2$, by the law of large numbers over the $n_\ell$ rows. The alignment is the ratio,

which is at least 1/2 whenever the fan-out is at least the fan-in, the condition Claim 1 states. For a square layer the input gets half of the maximum stretch, and a constant factor is invisible to $\Theta$. For the output layer of a wide network the ratio is small: with fan-out 1 and fan-in $n$ the matrix is a single row, and a random input is almost orthogonal to it, with alignment about $1/\sqrt n$, a number the section on standard parametrization uses.

At the defaults (256 by 256) the measured spectral norm is within a few percent of $\sigma(\sqrt m + \sqrt n) = 2$ and the initial-weight alignment is about 0.5. With both claims in hand, the features at initialization satisfy

and the recursion carries Desideratum 1 from the input to the last hidden layer. A worked case: a 1024 to 1024 hidden layer whose input has entries of size 1, so $\|h_{\ell-1}\|_2 = 32$. With $\|W_\ell\|_* = 1$, the features have norm about 16 (half the bound), entries about 0.5. An update with $\|\Delta W_\ell\|_* = 1$ built from this input changes the features by exactly 32 in norm, entries about 1. Double the width to 2048 and the same condition gives norms of about 23 and 45: entries unchanged.

The paper summarizes the argument in two observations. Updates are low rank and aligned with the inputs that produced them, so the spectral norm, the stretch of a perfectly aligned vector, is the size that predicts their effect. And working in spectral norms of weights and updates, instead of directly in initialization scales and learning rates, turns the scaling analysis into the three-line bound above.

More steps, nonlinearities, batches, Adam

The derivation above covers one step of one example through a linear network. Section 3.2 extends it one complication at a time, each with an assumption that Appendix C checks on a deep MLP.

More steps. After the first step the weights are $W_\ell + \Delta W_\ell$. Assumption 1 says the update does not cancel the initial weights or the initial features, so the new weight matrix still has spectral norm $\Theta(\sqrt{n_\ell/n_{\ell-1}})$ and the new features still have norm $\Theta(\sqrt{n_\ell})$. The second step then looks like the first. Cancellation of two high-dimensional objects needs a special reason; the paper notes that adding a little randomness to the learning rate removes any accidental one.

Nonlinearities. With $h'_\ell = \phi(h_\ell)$ the layer input becomes the post-activation $h'_{\ell-1}$, and the update is still the outer product of a gradient and that input, so (7) holds with $h'$ in place of $h$. Assumption 2 asks that the activation keep the scale, $\|h'_\ell\|_2 = \Theta(\|h_\ell\|_2)$. For ReLU on roughly zero-mean preactivations the ratio is about $1/\sqrt 2$, since half the entries survive; it would fail only if nearly every preactivation were negative.

Batches. With a batch of $B$ examples the update is the average of $B$ rank-one updates, so it has rank up to $B$ and cannot align perfectly with every input. For input $x_i$, one of the $B$ terms is exactly aligned with it; Assumption 3 says the other terms do not cancel that one, and Assumption 4 says $B$ does not grow with width. Together they give alignment of order 1, which is all the scaling argument needs.

The paper also measures something stronger: batch updates stay close to rank one even when $B$ is large. The measure is the stable rank, $\|\Delta W\|_F^2 / \|\Delta W\|_*^2$, the sum of the squared singular values divided by the largest one. It is 1 for a rank-one matrix and at most the ordinary rank, and it counts how many directions carry a comparable share of the matrix. On CIFAR-10 at width 300, the paper finds it below 10 at every batch size up to $10^4$, far under the maximum of 300. The figure repeats the measurement on a small random network.

At $B = 1024$ the ReLU gradient has stable rank 1.03 and the tanh gradient 4.17, while 1024 independent outer products reach 53.75 and are still climbing toward about 64, a quarter of the width, which is what a square Gaussian matrix gives. The real gradients stay low because the $B$ terms share directions. The paper's Appendix A shows that both the stacked forward vectors and the stacked backward gradients have low stable rank. For ReLU there is a concrete source: every post-ReLU vector is nonnegative, so all of them share a component along the all-ones direction, which for Gaussian preactivations carries $1/\pi \approx 32\%$ of each vector's squared length. Low stable rank also lets the Frobenius norm of a gradient stand in for its spectral norm, which the implementation section uses.

Adam. Adam turns each gradient entry into an update entry with an entrywise function. With both decay rates set to zero it is SignSGD, the update $-\eta\,\mathrm{sign}(\nabla_W \mathcal{L})$. The paper's Section 3.3 argues, through Tensor Programs in Appendix B, that entrywise processing keeps the updates low rank, so the same condition applies. For SignSGD at batch size one this can be checked by hand, and the check reproduces the μP learning rates for Adam. The gradient is $g h^\top$, and the sign of a product is the product of signs, so the update is $-\eta\, \mathrm{sign}(g)\,\mathrm{sign}(h)^\top$, still rank one. Its spectral norm is $\eta\sqrt{n_\ell}\sqrt{n_{\ell-1}}$, the product of the lengths of two vectors of $\pm 1$ entries. Setting that equal to $\sqrt{n_\ell/n_{\ell-1}}$ gives

for every layer: $1/n$ for hidden and output layers, and a width-independent constant for the input layer, whose fan-in is the fixed input dimension. These are the Adam learning rates in Table 3 of Tensor Programs V.

From norms to initialization scales and learning rates

The most direct way to satisfy Condition 1 is to normalize. Sample a matrix $W'_\ell$ with any standard initializer and rescale it to the target spectral norm, and rescale each gradient the same way:

with width-independent constants $\sigma$ and $\eta$. Computing a spectral norm every step is expensive, but because gradients have low stable rank the Frobenius norm is within a small constant of it and costs one pass over the entries:

# eq. (17) without the constants: rescale the raw gradient G of an
# (fan_out x fan_in) weight so its spectral norm is sqrt(fan_out / fan_in).
# The Frobenius norm stands in for the spectral norm because G has low
# stable rank (next section), and it costs one pass over G.
G = W.grad
W -= eta * math.sqrt(fan_out / fan_in) * G / G.norm()    # G.norm() is Frobenius

The paper's main route is to avoid normalizing at all, by working out in advance how the norms scale and choosing each layer's initialization standard deviation $\sigma_\ell$ and SGD learning rate $\eta_\ell$ to match.

Initialization. For a Gaussian matrix with entry standard deviation $\sigma_\ell$, $\|W_\ell\|_* \approx \sigma_\ell(\sqrt{n_\ell} + \sqrt{n_{\ell-1}})$. Setting this to $\sqrt{n_\ell / n_{\ell-1}}$ and keeping only the larger of the two square roots inside the sum gives the first half of Parametrization 1. When fan-out is at least fan-in the larger root is $\sqrt{n_\ell}$ and $\sigma_\ell = 1/\sqrt{n_{\ell-1}}$, the familiar fan-in initialization. When fan-out is smaller, the fan-in term dominates the spectral norm and the standard deviation picks up an extra factor $\sqrt{n_\ell/n_{\ell-1}}$. For ReLU networks the paper notes that the constant in front is usually $\sqrt 2$, as in Kaiming initialization.

Learning rate. The update is $-\eta_\ell \nabla_{W_\ell}\mathcal{L}$, so its spectral norm is $\eta_\ell$ times that of the gradient, and the size of the gradient has to be found. The paper gets it from the loss. A properly sized update moves the output by order one and so changes the loss by order one. To first order that change is the inner product of the update and the gradient, and because the update is a multiple of the gradient and both are low rank,

With $\Delta\mathcal{L} = \Theta(1)$ and $\|\Delta W_\ell\|_* = \Theta(\sqrt{n_\ell/n_{\ell-1}})$, the gradient must have the reciprocal size,

and the learning rate is the ratio of the two. Parametrization 1, in full:

The gradient size in (20) can be checked directly for a square hidden layer. Its gradient is $g\,h^\top$ with $\|h\|_2 = \sqrt n$. Under this parametrization the output layer's weights have entries of size $1/n$, and backpropagating an order-one output error through them gives $g$ entries of size $1/n$, so $\|g\|_2 = 1/\sqrt n$ and $\|g h^\top\|_* = 1$, matching (20) with $n_\ell = n_{\ell-1}$. The learning rate is then 1.

Applied to the shape of the paper's experiment (a 3072-pixel image input, two hidden layers of 4096 and one output), with a $\sqrt 2$ initialization constant and a base learning rate of 0.1, the rule gives:

import math

def spectral_init_and_lr(fan_in, fan_out, sigma0=math.sqrt(2), eta0=0.1):
    """Parametrization 1 for SGD. One rule for every weight matrix."""
    # init std: gives ||W||_* on the order of sqrt(fan_out / fan_in)
    std = sigma0 / math.sqrt(fan_in) * min(1.0, math.sqrt(fan_out / fan_in))
    # SGD learning rate: gives ||dW||_* on the order of sqrt(fan_out / fan_in)
    lr = eta0 * fan_out / fan_in
    return std, lr

# the paper's experiment shape, 3072 -> n -> n -> 1, at n = 4096
for fan_in, fan_out in [(3072, 4096), (4096, 4096), (4096, 1)]:
    print(fan_in, fan_out, spectral_init_and_lr(fan_in, fan_out))
# prints, rounded:
# 3072 4096  std 0.0255    lr 0.133      input layer
# 4096 4096  std 0.0221    lr 0.1        hidden layer
# 4096 1     std 0.000345  lr 2.44e-05   output layer

The three layers come out as three different cases of one formula. The input layer widens (3072 to 4096), so its learning rate is $4096/3072 \approx 1.33$ times the base. The hidden layer is square: fan-in initialization, base learning rate. The output layer narrows from 4096 to 1, so its initialization carries the extra $\sqrt{1/4096} = 1/64$ and its learning rate is the base divided by 4096. Doubling the hidden width to 8192 doubles the input layer's learning rate, leaves the hidden layer's alone, and halves the output layer's.

Set $n_0 = n_L = 1$ and make all hidden widths equal, and these formulas reproduce the μP table of Tensor Programs V for SGD: learning rates proportional to fan-out, 1 and 1/fan-in for the input, hidden and output layers, and initialization variances 1/fan-in, 1/fan-in and 1/fan-in². The original treats the three layer types separately. Here they are one formula, and it also covers hidden layers of unequal width and scaling of the input and output dimensions.

SP, NTP and μP against the condition

Most networks are trained in what the paper calls standard parametrization (SP): Kaiming, Xavier or LeCun initialization, which all scale the standard deviation roughly as $1/\sqrt{n_{\ell-1}}$, and one learning rate for every layer,

SP's initialization matches Parametrization 1 at every layer whose fan-out is at least its fan-in and is too large wherever fan-out is smaller, which includes the output layer. Its single learning rate matches only at square layers; at the input layer Parametrization 1 asks for a rate that grows with width. At the output layer, SP gives a row of $n$ entries of size $1/\sqrt n$, with spectral norm about 1 against a target of $1/\sqrt n$. At initialization this does no harm, because the random hidden vector has alignment $1/\sqrt n$ with that row and the output comes out of order 1. But training aligns the last hidden layer's features with the output weights (the paper's Figure 2C measures the alignment rising to order 1 under μP), and an aligned input receives the full spectral norm: an output of order $\sqrt n$. The same oversized output weights make the gradients reaching the hidden layers $\sqrt n$ times too large, so with a fixed learning rate the hidden updates are too large as well.

The neural tangent parametrization shares SP's initialization but divides each learning rate by the fan-in:

For a hidden layer, the gradient is $\sqrt n$ times too large (the output layer again) and the learning rate is $n$ times smaller than Parametrization 1's value of 1, so the update's spectral norm is $\sqrt n / n = 1/\sqrt n$ of what the condition asks for. The update vanishes as the network widens and the features stop moving. The output layer still learns, so the network output fits the data through fixed features, which is the kernel regime. The paper notes that rescaling the NTP network's output, with a matching change to the global learning rate, converts it into μP.

The figure runs all three on one network. It is a ReLU MLP with a 32-dimensional input, two hidden layers of width $n$ and one output, taking one SGD step on one example with the loss $-f$ so the output gradient is the same at every width. The three rules agree exactly at the base width 64 and differ only in how they scale away from it.

From width 64 to 4096 (64 times wider, so $\sqrt{64} = 8$): under μP the relative feature change is 0.128 at both ends. Under NTP it falls to 0.0161, eight times smaller. Under SP it rises to 0.61, and the relative spectral size of the hidden update grows 8.1 times. In output change, μP and NTP agree: both move the output by about 2 at every width, so both train the network, and only μP also moves the features. SP moves it by 2.07 at width 64 and 99.9 at 4096, a factor of 48, and a run of many such steps diverges. Under SP the output layer supplies about half of that change at width 64, and its own contribution is predicted to grow 64-fold: with the loss $-f$, that layer's update is $\eta h$, parallel to the last hidden vector $h$, so it changes $f$ by $\eta \|h\|_2^2$, a sum of $n$ order-one terms that grows in proportion to $n$. The total grows by less, a factor of 48, because the hidden layers supply the other half of the 2.07 at width 64 and their part grows more slowly with width.

Under SP the relative Frobenius change of the hidden weights is flat in width (0.0092 to 0.0093) while the relative spectral change grows as $\sqrt n$; under μP the spectral change is flat and the Frobenius change shrinks as $1/\sqrt n$. The paper uses this to object to optimizers that set each layer's step as a fixed fraction of its weights in Frobenius norm, such as LARS and LAMB. The weights have stable rank of order $n$ and the updates of order 1, so a fixed Frobenius ratio produces updates whose spectral size grows as $\sqrt{\min(n_\ell, n_{\ell-1})}$. The corrected rule divides by that factor: $\|\Delta W_\ell\|_F \propto \|W_\ell\|_F / \sqrt{\min(n_\ell, n_{\ell-1})}$. The paper traces the error to an earlier analysis by one of its own authors (Bernstein et al., 2020), which assumed weights and their updates have the same conditioning.

The paper's experiment

Section 6 trains three-layer ReLU MLPs without biases, input 3072, two hidden layers of width $n$ from 16 to 4096, one output, on 200 CIFAR-10 images of airplanes and automobiles with $\pm 1$ targets. Each width is trained twice, under μP and under NTP, full batch for $10^4$ steps with global learning rate 0.1, which brings the training loss below 0.01 at every width. The two parametrizations are defined to coincide at width 1. After training the paper compares the final network to the initial one:

is the relative change of the second hidden layer's preactivations (panel A). Panels B and D give the relative change of the hidden weight matrix $W_2$ in spectral and in Frobenius norm, and panel C the final-layer alignment,

Under μP, the relative feature change stays near 0.6 across the whole width range, the relative spectral change of $W_2$ stays near 0.6, and the final-layer alignment grows from its initial $1/\sqrt n$ to about 0.25 at every width. Under NTP all three fall with slope $-1/2$ on the log-log plot, the feature change reaching about 0.05 at width 4096. In Frobenius norm even μP's weight change falls, with slope $-1/2$, and NTP's with slope $-1$. So a network whose features change by 60% can show a relative Frobenius weight change of a few percent. The paper puts earlier plots of wide networks' small weight changes (Lee et al., 2019, Figure 1) in that context.

The authors add a caveat about their own result: the theory covers a few gradient steps relative to the width, and these runs take $10^4$ steps, more than the width in every case. They report being surprised that the measured slopes match the predicted ones this closely.

What is proven and what is assumed

The main text is a scaling argument, not a theorem. It uses Claims 1 and 2 and Assumptions 1 to 4, and its statements hold up to constants that $\Theta$ hides. Appendix C checks Assumptions 1 to 3 empirically on MLPs, and Assumption 4 only fixes the batch size at $B = \Theta(1)$. The rigorous version is Appendix B, which works in the Tensor Programs framework under the assumption that the architecture and optimizer can be written as a tensor program. There, Theorem 2 states that under μP, for all but a measure-zero set of learning rates, Condition 1 holds at any fixed time in training once the width is large enough, and that μP is the only stable and faithful abcd-parametrization with this property. An abcd-parametrization sets four powers of width for each layer: a multiplier on the weights (a), the initialization scale (b), the learning rate (c) and a multiplier on the gradient before the optimizer's update rule is applied (d). Stable means features and output stay bounded as width grows; faithful means that update rule receives inputs of order 1. The proof rests on a lemma: $\|\Delta W\|_F / \|\Delta W\|_*$ stays of order 1 as width grows, which means updates keep a width-independent effective rank. Section 3.4 of the main text adds that exceeding the condition at any layer makes training blow up with width and cites Appendix B for the proof; Appendix B proves the uniqueness statement, not the blow-up.

The condition covers width only: depth, dataset size and learning-rate schedule are all inside the constants, so it says nothing about how hyperparameters should change when a network gets deeper. The derivation also assumes dense inputs with norm $\sqrt{n_0}$. For a language model's embedding layer, whose input is one-hot with norm 1, the $\sqrt{n_0}$ in the condition becomes 1. Appendix E unifies the two cases by measuring each dense vector in RMS norm, $\|v\|_2/\sqrt{m}$ for a vector of length $m$, and each sparse vector in the ordinary norm. The spectral norm induced by RMS norms on both sides of an $m \times n$ matrix is $\sqrt{n/m}\,\|A\|_*$, and in these units Condition 1 reads: every weight matrix and every update has norm of order 1. The paper prints the factor as $\sqrt{m/n}$, which is inverted; the Provenance panel gives the check. Biases fit the same scheme by treating a bias as a weight matrix from a constant input of width 1: initialization of order 1 (zero in practice) and learning rate proportional to $n_\ell$.

Two results in this family came after the paper. Everett et al. (2024) reported that other parametrizations, standard included, also transfer learning rates across width once each layer's learning rate is scaled separately. And spectral normalization of updates became an optimizer: Muon orthogonalizes each matrix update, so all its singular values are near 1, and scales it by $\sqrt{\max(1, \text{fan-out}/\text{fan-in})}$. Bernstein and Newhouse (2024) describe Shampoo, Adam and related optimizers as steepest descent under particular norms, and the Muon write-up credits that line of work.