The cleanest way I have found to understand Tensor Programs IV is to see μP as the critical width scaling at which every layer can still move features by an order-one amount without making the network blow up. Tensor Programs V then notices that, in these normalized coordinates, a master hyperparameter such as the learning rate keeps the same dynamical meaning as width changes. That is the mechanism behind μTransfer.
Companion note: Deep information propagation explains the initialization-side variance and gradient calculations, and why they do not by themselves select a feature-learning parametrization.
1. The big picture
Schematically, the story is
\[ \boxed{\text{TP IV: find the critical infinite-width feature-learning scaling}} \;\longrightarrow\; \boxed{\text{TP V: use those normalized coordinates for hyperparameter transfer}}. \]There is a direct analogy with the depth story in TP VI. In width, the key distinction is between sums of fresh random contributions and sums of learned correlated contributions. The former accumulate like \(\sqrt n\); the latter can accumulate like \(n\).
2. The two widthwise summation laws
Suppose \(x\in\mathbb{R}^n\) has order-one coordinates.
Fresh Gaussian matrix: \(\sqrt n\) accumulation
If \(W\in\mathbb{R}^{n\times n}\) has independent zero-mean entries and is essentially independent of \(x\), then each coordinate of \(Wx\) is a random sum. If \(W_{ij}=O(n^{-1/2})\), then
\[ (Wx)_i =\sum_{j=1}^n W_{ij}x_j \sim \sqrt n\times n^{-1/2} =O(1). \]This is the familiar Xavier/LeCun/CLT rule.
Learned outer-product matrix: \(n\) accumulation
A one-sample SGD update to a hidden matrix has the form
\[ \Delta W\propto \delta x^\top. \]When this acts on the same feature vector \(x\),
\[ (\delta x^\top)x =\delta\,\|x\|^2. \]Since \(\|x\|^2=\Theta(n)\), this is an \(n\)-sized coherent accumulation. There is no \(\sqrt n\) cancellation because the update is built from the very feature vector it later acts on.
3. A three-layer MLP
Consider a two-hidden-layer MLP of width \(n\):
\[ h_1=W_1\xi,\qquad x_1=\phi(h_1), \] \[ h_2=W_2x_1,\qquad x_2=\phi(h_2), \] \[ f=W_3x_2. \]Take
\[ W_1\in\mathbb{R}^{n\times d},\qquad W_2\in\mathbb{R}^{n\times n},\qquad W_3\in\mathbb{R}^{1\times n}, \]where the input dimension \(d\) is fixed while \(n\to\infty\).
The μP design criterion is: choose initialization and learning-rate scales so that hidden activations stay order one, the output does not blow up, and every layer can still induce an order-one change in the features it controls.
4. Why the input weights are order one
For one coordinate of the first preactivation,
\[ (h_1)_i=\sum_{j=1}^d (W_1)_{ij}\xi_j. \]There are only \(d=O(1)\) terms. No width-sized sum appears. Therefore we simply need the entries of \(W_1\) to stay order one with respect to \(n\). With standard fan-in normalization,
\[ \boxed{(W_1)_{ij}\sim\mathcal N\!\left(0,\frac1d\right)}. \]Then each coordinate of \(h_1\) is \(O(1)\).
5. Why hidden weights are order \(n^{-1/2}\)
The second layer sees a width-\(n\) input:
\[ (h_2)_i=\sum_{j=1}^n (W_2)_{ij}(x_1)_j. \]At initialization, \(W_2\) is a fresh random matrix and is approximately independent of \(x_1\). Therefore these \(n\) terms add like a random walk. To obtain an order-one preactivation, take
\[ (W_2)_{ij}=O(n^{-1/2}), \]equivalently
\[ \boxed{(W_2)_{ij}\sim\mathcal N\!\left(0,\frac1n\right)}. \]This part of μP is ordinary fan-in initialization.
6. Why output weights are order \(n^{-1}\)
The surprising layer is the readout:
\[ f=\sum_{i=1}^n (W_3)_i(x_2)_i. \]At initialization alone, the usual CLT argument would suggest \((W_3)_i=O(n^{-1/2})\). That indeed produces an order-one random output initially.
But feature-learning theory cares about the training trajectory, not only the first forward pass. After training begins, \(W_3\) and \(x_2\) become correlated. A correlated sum can add like \(n\), so the critical scale for an output vector that must remain stable under training is
\[ \boxed{(W_3)_i=O(n^{-1})}. \]In the convenient TP V formulation,
\[ \boxed{(W_3)_i\sim\mathcal N\!\left(0,\frac1{n^2}\right)}. \]Because initialization is still random and centered, this means the initial logit is actually smaller:
\[ f_0\sim \sqrt n\,\frac1n=\frac1{\sqrt n}\longrightarrow 0. \]This is intentional. TP IV proves, within its class of stable nontrivial feature-learning parametrizations, that the infinite-width initial output must vanish.
7. Why the output-layer SGD learning rate is \(\eta/n\)
Let
\[ \chi=\frac{\partial\ell}{\partial f}=O(1). \]The output gradient is
\[ \nabla_{W_3}\ell=\chi x_2^\top. \]Each coordinate is therefore \(O(1)\). If we used an order-one learning rate, then \(\Delta W_3=O(1)\) coordinatewise, and
\[ \Delta f =\Delta W_3x_2 \propto x_2^\top x_2 =\Theta(n). \]The output would blow up with width.
The critical choice is therefore
\[ \boxed{\eta_{W_3}=\frac{\eta}{n}}. \]Then
\[ \Delta W_3=O(n^{-1}), \]and because the update is aligned with \(x_2\),
\[ \Delta W_3x_2 =-\frac{\eta}{n}\chi\|x_2\|^2 =O(\eta). \]The readout moves by an order-one amount, independent of width.
8. Why the hidden-layer SGD learning rate is \(\eta\)
Backpropagate through the readout:
\[ \delta_2 \equiv \frac{\partial\ell}{\partial h_2} =\chi W_3^\top\odot\phi'(h_2). \]Since \(W_3=O(n^{-1})\),
\[ \boxed{\delta_{2,i}=O(n^{-1})}. \]The gradient of the hidden matrix is the outer product
\[ \nabla_{W_2}\ell=\delta_2x_1^\top, \]so its coordinates are also \(O(n^{-1})\). If we choose an order-one SGD learning rate,
\[ \boxed{\eta_{W_2}=\eta}, \]then
\[ \Delta W_2=O(n^{-1}). \]Entrywise, this is smaller than the initialization \(W_{2,0}=O(n^{-1/2})\). But the relevant quantity is not the coordinatewise movement of the matrix; it is the induced feature movement:
\[ \Delta W_2x_1 =-\eta\,\delta_2(x_1^\top x_1). \]Since \(\delta_2=O(n^{-1})\) and \(x_1^\top x_1=\Theta(n)\),
\[ \boxed{\Delta W_2x_1=O(\eta)}. \]9. Why the input-layer SGD learning rate is \(\eta n\)
Backpropagate one layer further:
\[ \delta_1 =W_2^\top\delta_2\odot\phi'(h_1). \]At leading order, \(W_2\) has entries \(O(n^{-1/2})\), while \(\delta_2\) has entries \(O(n^{-1})\). One term in the sum therefore has size \(n^{-3/2}\). Summing \(n\) fresh random terms gives a CLT factor \(\sqrt n\), so
\[ \delta_1=O(n^{-1}). \]Thus
\[ \nabla_{W_1}\ell=\delta_1\xi^\top \]also has coordinate size \(O(n^{-1})\).
But unlike \(W_2\), the input matrix multiplies a fixed-dimensional vector \(\xi\in\mathbb{R}^d\). There is no width-sized inner product to amplify a small update. If we used an order-one LR,
\[ \Delta W_1\xi=O(n^{-1})\to0, \]and the first layer would freeze.
To keep its feature movement order one, the update must be amplified by \(n\):
\[ \boxed{\eta_{W_1}=\eta n}. \]Then
\[ \Delta W_1=O(1), \qquad \boxed{\Delta W_1\xi=O(\eta)}. \]10. The whole scaling in one table
For this two-hidden-layer MLP, one convenient SGD form of μP is
\[ W_1\sim\mathcal N(0,1/d),\qquad W_2\sim\mathcal N(0,1/n),\qquad W_3\sim\mathcal N(0,1/n^2), \]with
\[ \eta_{W_1}=\eta n,\qquad \eta_{W_2}=\eta,\qquad \eta_{W_3}=\frac{\eta}{n}. \]| Layer | Init coordinate size | Gradient size | SGD LR | Update size | Induced change |
|---|---|---|---|---|---|
| \(W_1:n\times d\) | \(1\) | \(1/n\) | \(\eta n\) | \(1\) | \(\Delta W_1\xi=O(\eta)\) |
| \(W_2:n\times n\) | \(1/\sqrt n\) | \(1/n\) | \(\eta\) | \(1/n\) | \(\Delta W_2x_1=O(\eta)\) |
| \(W_3:1\times n\) | \(1/n\) | \(1\) | \(\eta/n\) | \(1/n\) | \(\Delta W_3x_2=O(\eta)\) |
The design principle is therefore
\[ \boxed{\Delta h_1,\;\Delta h_2,\;\Delta f=O(\eta)\quad\text{for every width }n.} \]11. Why standard parametrization fails
Standard parametrization uses roughly
\[ W_1=O(1),\qquad W_2=O(n^{-1/2}),\qquad W_3=O(n^{-1/2}). \]The readout is the problem. Since \(\nabla W_3=O(1)\), an order-one global LR makes
\[ \Delta W_3=O(1), \]and therefore
\[ \Delta f\sim n. \]The network blows up as width grows.
One can stabilize it by shrinking the global LR to \(O(1/n)\). But then hidden feature movement becomes too small. Roughly, the second-layer update becomes \(O(n^{-3/2})\), so
\[ \Delta W_2x_1=O(n^{-1/2})\to0. \]The network enters a lazy/kernel regime.
So standard parametrization presents a bad choice:
\[ \boxed{\text{large LR} \Rightarrow \text{unstable output}}, \] \[ \boxed{\text{small LR} \Rightarrow \text{frozen features}}. \]TP IV's dynamical dichotomy makes this precise in its class of abc-parametrizations: stable nontrivial limits are either feature-learning or kernel-like, and μP lies on the feature-learning side.
12. Why it is called maximal update
The word “maximal” is literal. For each layer, ask:
How large can the update be, as a function of width, without making some activation or the output diverge?
μP sits at that boundary. The induced feature changes remain order one:
\[ \Delta W_1\xi=\Theta(1), \qquad \Delta W_2x_1=\Theta(1), \qquad \Delta W_3x_2=\Theta(1), \]for suitable nondegenerate training routines. No layer is asymptotically frozen, and making the relevant update parametrically larger would violate stability.
TP IV formalizes a stronger statement in its parametrization class: μP is the unique stable choice for which all layers learn features maximally in the specified sense.
13. How this matches TP IV's abc-parametrization
TP IV writes
\[ W_\ell=n^{-a_\ell}w_\ell, \]initializes the trainable parameter as
\[ (w_\ell)_{ij}\sim\mathcal N(0,n^{-2b_\ell}), \]and uses a global SGD learning rate \(\eta n^{-c}\).
For μP,
\[ c=0,\qquad b_\ell=\frac12, \]while
\[ a_1=-\frac12, \qquad a_{\text{hidden}}=0, \qquad a_{\text{out}}=\frac12. \]Thus
\[ W_1=\sqrt n\,w_1,\qquad W_2=w_2,\qquad W_3=\frac1{\sqrt n}w_3, \]with every \(w\)-coordinate initialized at scale \(n^{-1/2}\). This gives the effective initialization scales
\[ W_1=O(1),\qquad W_2=O(n^{-1/2}),\qquad W_3=O(n^{-1}). \]If \(W=s w\) and SGD updates \(w\) with LR \(\eta\), then
\[ \frac{\partial\ell}{\partial w} =s\frac{\partial\ell}{\partial W}, \]so
\[ \Delta W =s\Delta w =-\eta s^2\frac{\partial\ell}{\partial W}. \]The effective LR on \(W\) is therefore \(\eta s^2\). Plugging in the three multipliers gives
\[ s_1=\sqrt n\Rightarrow \eta_{W_1}=\eta n, \] \[ s_2=1\Rightarrow \eta_{W_2}=\eta, \] \[ s_3=n^{-1/2}\Rightarrow \eta_{W_3}=\eta/n. \]So the abstract abc formulation in IV and the practical per-layer recipe in V are the same scaling written in different coordinates.
14. Equivalent ways to write μP
There is no single sacred literal representation of μP. Width factors can be moved between
- initialization scale,
- explicit parameter multipliers in the forward pass, and
- learning-rate scale,
provided the resulting function and training dynamics are equivalent.
For example, instead of literally drawing \(W_3\) with variance \(1/n^2\), one can initialize an underlying parameter at the more familiar \(1/n\) variance and place an explicit \(1/\sqrt n\) multiplier in the forward map. The optimizer scaling changes correspondingly.
TP V gives several equivalent μP tables for exactly this reason.
15. From TP IV to TP V: why μTransfer is plausible
Now the practical insight of TP V becomes almost inevitable.
Under μP, one master learning-rate parameter \(\eta\) induces
\[ \eta_{W_1}=\eta n, \qquad \eta_{W_2}=\eta, \qquad \eta_{W_3}=\eta/n, \]and therefore
\[ \Delta h_1=O(\eta),\qquad \Delta h_2=O(\eta),\qquad \Delta f=O(\eta), \]at every width.
So the number \(\eta\) has the same dynamical meaning at width 128 and width 8192: it controls an order-one normalized step in the limiting feature-learning dynamics.
In standard parametrization, the same raw LR can correspond to a width-dependent effect such as \(\Delta f\sim \eta n\). Then the semantic meaning of “learning rate \(10^{-3}\)” changes with width, and there is no reason its optimum should stay fixed.
The toy-model intuition
TP V illustrates this with a CLT example. Suppose
\[ F_n(c)=\mathbb E f\!\left(c\sum_{i=1}^n x_i\right). \]The raw coordinate \(c\) is badly scaled. Write instead
\[ c=\frac{\alpha}{\sqrt n}. \]Then the family of objectives can converge to a fixed limiting function of \(\alpha\):
\[ F_n(\alpha/\sqrt n)\longrightarrow G_\infty(\alpha). \]If this convergence is sufficiently well behaved, then the optimizing coordinate stabilizes:
\[ \alpha_n^\star\longrightarrow \alpha_\infty^\star. \]μP plays the role of identifying the analogous normalized coordinates for a whole neural network.
16. What actually gets transferred
One does not generally copy the literal per-layer learning rates between a proxy and a target model. One copies the width-independent master hyperparameter.
If a base model has width \(n_0\) and a target model has width \(n\), define
\[ \widetilde n=\frac{n}{n_0}. \]A base-shape version of the SGD rule can be written as
\[ \eta_{W_1}=\eta\widetilde n, \qquad \eta_{W_2}=\eta, \qquad \eta_{W_3}=\frac{\eta}{\widetilde n}. \]At \(n=n_0\), all width multipliers equal one, so the chosen base model can be exactly backward-compatible with a conventional implementation. As width changes, μP specifies how the optimizer and parameter multipliers must deform.
Thus μP is better understood as a scaling rule than as an absolute prescription at one fixed width.
17. Why an arbitrary infinite-width limit is not enough
A standard network with a sufficiently small learning rate can also have a perfectly legitimate infinite-width limit. The NTK regime is the canonical example.
But in a kernel limit, hidden features freeze. Then some layerwise learning rates cease to influence the infinite-width dynamics even though they still matter at finite width. In that situation the limiting model has lost the same hyperparameter semantics as the finite network.
This is a deeper reason maximal feature learning matters for transfer: every parameter tensor continues to have an order-one effect in the normalized limiting dynamics.
TP V's empirical claim is that these normalized coordinates make many important hyperparameter optima approximately stable across width. It also explicitly notes a remaining theoretical puzzle: the optimum often converges much faster than the full trained function or loss itself. In the paper's language, the HP optimum appears to behave like a coarse or macroscopic variable.
18. The common principle behind IV and VI
The width and depth theories can be aligned almost line by line.
| Width-μP | Depth-μP |
|---|---|
| random Gaussian action accumulates like \(\sqrt n\) | random residual increments accumulate like \(\sqrt L\) |
| learned correlated matrix action accumulates like \(n\) | learned correlated residual drift accumulates like \(L\) |
| hidden init scale \(n^{-1/2}\) | branch init scale \(L^{-1/2}\) |
| hidden learned update scale \(n^{-1}\) | learned branch contribution scale \(L^{-1}\) |
The common conceptual rule is
\[ \boxed{\text{initialization is controlled by CLT scaling; training is controlled by coherent/LLN scaling.}} \]That is the thread connecting TP IV, V, and VI.
19. Five equations to remember
The last three equations are the essence of μP: one master \(\eta\) produces the same order-one dynamical effect at every layer and every width. That is why the master hyperparameter has a chance to μ-transfer.
References
- Greg Yang and Edward J. Hu. Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks. ICML 2021. PMLR.
- Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022. arXiv:2203.03466.
- Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. ICLR 2024. arXiv:2310.02244.
These are expository working notes based on the Tensor Programs papers. The original papers should be taken as the source for precise assumptions, theorem statements, and proofs.