The cleanest way I have found to read Tensor Programs VI is as the depth analogue of the story in Tensor Programs IV and V. IV identifies the critical infinite-width parametrization that preserves feature learning; V turns that critical limit into widthwise hyperparameter transfer; VI asks for the corresponding critical scaling when the number of residual blocks tends to infinity.
Companion note: Deep information propagation derives residual variance, correlation, and gradient recursions, and distinguishes preservation of input geometry from feature diversity across depth.
1. The big picture
Schematically, the progression is
\[ \boxed{\text{TP IV: infinite width}} \;\longrightarrow\; \boxed{\text{TP V: widthwise }\mu\text{Transfer}} \;\longrightarrow\; \boxed{\text{TP VI: infinite depth}}. \]There is one important qualification. VI does not replace the infinite-width theory. Its analysis takes width to infinity first and depth second:
\[ n\to\infty \qquad\text{then}\qquad L\to\infty. \]The paper interprets this as the regime \(1\ll L\ll n\), which is also a useful mental model for why the widthwise Tensor Program calculus can be used before passing to a continuous depth coordinate.
2. The depth-scaling problem
Consider a residual network with \(L\) blocks. VI writes the residual stream in the generic form
\[ x^\ell = x^{\ell-1} + L^{-\alpha} g^\ell(x^{\ell-1};W^\ell), \qquad \ell=1,\dots,L, \]and lets the characteristic parameter-update size scale as
\[ \Delta W^\ell = O(L^{-\gamma}). \]The two exponents play different roles. The branch exponent \(\alpha\) controls how much each residual block contributes to the forward dynamics. The update exponent \(\gamma\) controls how quickly each block moves under training.
The question is then: which pair \((\alpha,\gamma)\) gives a depth limit that is stable, nontrivial, feature-learning, and not wastefully redundant?
3. Two summation laws
The whole scaling argument is driven by a distinction between two ways that terms add over depth.
Initialization: random increments add like \(\sqrt L\)
At initialization, different residual blocks contain fresh random weights. Very roughly, their centered contributions behave like random increments. One block contributes \(O(L^{-\alpha})\), so accumulating \(L\) of them produces
\[ \sqrt{L}\,L^{-\alpha}=L^{1/2-\alpha}. \]Therefore initialization remains stable only if
\[ \boxed{\alpha\ge \tfrac12}. \]The largest residual branch consistent with this stability condition is the boundary choice \(\alpha=1/2\).
Training: learned increments add coherently like \(L\)
A weight update of size \(L^{-\gamma}\) passes through a branch multiplier \(L^{-\alpha}\), so its direct contribution to the residual stream is of order
\[ L^{-\alpha-\gamma}. \]These learned contributions are not fresh independent noise: gradients in neighboring blocks are highly correlated. Their aggregate therefore scales like
\[ L\,L^{-\alpha-\gamma}=L^{1-\alpha-\gamma}. \]This is the second summation law, and it is the source of the training criticality condition.
4. The critical line \(\alpha+\gamma=1\)
The training contribution behaves differently on the two sides of the line \(\alpha+\gamma=1\):
| Scaling | Large-depth behavior |
|---|---|
| \(\alpha+\gamma<1\) | training updates accumulate too strongly and the dynamics become unstable; |
| \(\alpha+\gamma>1\) | the aggregate training effect vanishes, giving a trivial depth limit; |
| \(\alpha+\gamma=1\) | training produces an \(O(1)\), nontrivial change. |
Thus stability and nontriviality force
\[ \boxed{\alpha+\gamma=1}. \]Combining this with initialization stability leaves an entire family of valid critical limits:
\[ \alpha+\gamma=1, \qquad \frac12\le \alpha\le 1. \]Examples include \((1/2,1/2)\), \((3/4,1/4)\), and \((1,0)\). The main conceptual work of VI is explaining why these limits are not equally good.
5. Why \(1/L\) is not the end of the story
The choice
\[ (\alpha,\gamma)=(1,0) \]gives the familiar residual scaling
\[ x^\ell=x^{\ell-1}+\frac1L g^\ell(x^{\ell-1}). \]If \(\lambda=\ell/L\), this naturally suggests a smooth continuous-depth limit. In the simplest picture, differences over \(\varepsilon L\) layers scale as
\[ x^{(\lambda+\varepsilon)L}-x^{\lambda L}=O(\varepsilon), \]which is the regularity one expects from an ODE trajectory.
This limit can be stable, nontrivial, and feature-learning. But VI argues that it is still suboptimal because nearby layers become asymptotically too similar. The hidden-layer randomness gets washed out, making the depth trajectory smooth and locally redundant.
6. Why \(1/\sqrt L\) is special
Now set \(\alpha=1/2\). Over a window of \(\varepsilon L\) layers, the accumulated initialization noise has size
\[ \frac1{\sqrt L}\sqrt{\varepsilon L}=\sqrt{\varepsilon}. \]It therefore survives the depth limit. In the language of the paper, this gives maximal feature diversity: nearby layers differ on the rough scale
\[ \|x(\lambda+\varepsilon)-x(\lambda)\|\asymp \sqrt{\varepsilon}, \]rather than the smooth \(O(\varepsilon)\) scale.
This is the hallmark of Brownian increments. VI formalizes the distinction through a feature-diversity exponent: Depth-\(\mup\) has exponent \(1/2\), while the critical parametrizations with \(\alpha>1/2\) are smooth/redundant in the corresponding sense.
Since the critical line requires \(\alpha+\gamma=1\), choosing the largest initialization noise compatible with stability immediately forces
\[ \boxed{\alpha=\gamma=\frac12}. \]This is Depth-\(\mup\).
7. The SDE picture
The previous calculation gives a useful continuous-depth mnemonic. Let
\[ d\lambda=\frac1L. \]At Depth-\(\mup\), the fresh random part of one residual step is \(O(L^{-1/2})=O(\sqrt{d\lambda})\), while the training-induced drift of one step is \(O(L^{-1})=O(d\lambda)\). Schematically,
\[ dx_\lambda = \sigma_\lambda\,dB_\lambda + b_\lambda\,d\lambda. \]This equation should be read as intuition rather than as the literal full Tensor Program state equation. The actual limit tracks richer random-variable/operator objects. But the separation into a \(\sqrt{d\lambda}\) Gaussian term and a \(d\lambda\) drift term is exactly the right structural picture.
| Branch scaling | Continuous-depth character |
|---|---|
| \(1/L\) | smooth, ODE-like trajectory; |
| \(1/\sqrt L\) | rough, diffusion-like trajectory retaining hidden-layer randomness. |
8. SGD versus Adam
The invariant prescription is not simply “multiply the learning rate by \(L^{-1/2}\).” The fundamental condition is
\[ \boxed{\Delta W^\ell=O(L^{-1/2}).} \]How this translates into a raw learning-rate rule depends on the optimizer.
For SGD, the backpropagated gradient already contains the residual multiplier \(L^{-1/2}\). An \(O(1)\) master learning rate therefore gives
\[ \Delta W^\ell_{\mathrm{SGD}}=O(L^{-1/2}). \]For Adam, coordinatewise normalization removes the overall scale of the gradient, so the learning rate itself must supply the factor:
\[ \eta_{\mathrm{Adam}}\propto L^{-1/2}. \]9. Where Tensor Programs enter
The proof strategy has two asymptotic stages.
First: width \(n\to\infty\)
The Tensor Programs machinery replaces high-dimensional vectors by random variables representing their coordinate distributions. Schematically,
\[ x\in\mathbb{R}^n \quad\rightsquigarrow\quad |x\rangle, \qquad \frac{x^\top y}{n}\to \mathbb{E}[|x\rangle |y\rangle]. \]This is the same basic simplification that underlies the width theory in IV.
Second: depth \(L\to\infty\)
One introduces fractional depth \(\lambda=\ell/L\in[0,1]\). In the linear warm-up, discrete coefficient and covariance arrays—denoted by \(\Gamma\) and \(C\) in the paper—are rescaled into functions of continuous depth variables. Their finite-depth recursions then converge to integral equations.
The conceptual chain is therefore
\[ \text{finite neural network} \xrightarrow[n\to\infty]{} \text{random-variable Tensor Program} \xrightarrow[L\to\infty]{} \text{continuous-depth random process}. \]10. Hyperparameter transfer across depth
Tensor Programs V turns the correct width parametrization into a practical statement: in \(\mup\), the hyperparameter landscape stabilizes with width, so one can tune on a smaller proxy model and transfer to a larger target model.
VI argues for the depth analogue. With
\[ x^\ell=x^{\ell-1}+\frac{a}{\sqrt L}g^\ell(x^{\ell-1};W^\ell), \]and \(O(L^{-1/2})\) parameter updates, the constants such as the block multiplier \(a\) and the master learning-rate parameter can approach depth-independent optima. The paper's experiments train residual networks with up to \(2^{10}=1024\) blocks and show much more stable learning-rate landscapes under Depth-\(\mup\) than under standard depth scalings.
A particularly useful lesson from Appendix I is that “a limit exists” does not imply “hyperparameters transfer.” The \(1/L\) ODE scaling has a legitimate feature-learning limit but is still suboptimal in the paper's classification because it loses feature diversity.
11. Mean subtraction
The main nonlinear setup in VI uses centered residual branches:
\[ x^\ell =x^{\ell-1}+L^{-\alpha}\operatorname{MS}\!\left(\phi(W^\ell x^{\ell-1})\right), \]where
\[ \operatorname{MS}(z)=z-\frac1n\langle z,\mathbf{1}\rangle\mathbf{1}. \]The issue already appears at initialization: centered preactivations need not give centered activations. For example, if \(Z\sim\mathcal{N}(0,1)\), then \(\mathbb{E}[\operatorname{ReLU}(Z)]=1/\sqrt{2\pi}>0\). Thus even zero-mean random weights produce a positive average residual contribution after ReLU. Controlling the size of each branch is not enough, because this shared offset accumulates coherently across layers.
For a simple scaling picture, suppose the unscaled branches have bounded statistics and an approximately constant mean \(m>0\), and write \(g^\ell=m\mathbf{1}+\varepsilon^\ell\), with \(\varepsilon^\ell\) representing centered random fluctuations. With the \(1/\sqrt L\) multiplier, the mean accumulates to \(m\sqrt L\), while the fresh centered fluctuations accumulate to order one. Shrinking the multiplier to \(1/L\) would keep this mean bounded, but would also shrink the random fluctuations to \(O(L^{-1/2})\), losing the randomness that supports feature diversity. The illustrative decomposition is
\[ \frac1{\sqrt L}\sum_{\ell=1}^L g^\ell =\underbrace{m\sqrt L\,\mathbf{1}}_{\text{coherent mean}} +\underbrace{\frac1{\sqrt L}\sum_{\ell=1}^L\varepsilon^\ell}_{\text{order-one random fluctuations}}. \]Mean subtraction removes the average across feature coordinates, separately for each input and layer, so the larger \(1/\sqrt L\) multiplier can preserve these fluctuations without accumulating the shared offset. This is the centering operation used in LayerNorm, without the variance normalization. It does not remove all learned drift: training can still produce coordinated changes in centered feature directions. At Depth-\(\mup\), a branch change of order \(L^{-1/2}\) receives another \(L^{-1/2}\) from the residual multiplier, giving an \(O(1/L)\) contribution per block that can sum to an \(O(1)\) learning effect.
12. The block-depth limitation
The clean optimality result is for residual blocks containing one trainable matrix, schematically
\[ g^\ell(x)=\phi(W^\ell x). \]This is an important caveat for modern architectures. A Transformer MLP block, for example, contains a composition such as
\[ W_2\phi(W_1x), \]so the block itself has depth at least two.
Appendix K explains the problem. If both parameter changes are \(O(L^{-1/2})\), then products such as
\[ \Delta W_2\,\Delta W_1 \]are lower order and disappear as \(L\to\infty\). The block therefore becomes linearized with respect to the learned parameter changes: terms such as \(W_{2,0}\Delta W_1\) and \(\Delta W_2 W_{1,0}\) survive, but the multiplicative interaction between the two learned updates does not.
The paper argues that this creates a genuine defect for block depth \(\ge 2\) and observes associated hyperparameter shifts experimentally. So VI should not be read as a complete final scaling law for arbitrarily deep modern Transformers; rather, it exposes a new obstruction that appears once a residual block contains several interacting trainable layers.
13. Five equations to remember
Equivalently, Depth-\(\mup\) uses
\[ \boxed{ x^{\ell+1}=x^\ell+\frac1{\sqrt L}g^\ell(x^\ell), \qquad \Delta W^\ell=O\!\left(\frac1{\sqrt L}\right). } \]The corresponding continuous-depth mnemonic is
\[ dx_\lambda \sim \underbrace{dB_\lambda}_{\text{initialization diversity}} + \underbrace{b_\lambda\,d\lambda}_{\text{feature-learning drift}}. \]That is the central mental model I want to retain from Tensor Programs VI.
References
- Greg Yang and Edward J. Hu. Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks. ICML 2021. PMLR.
- Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022. arXiv:2203.03466.
- Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. ICLR 2024. arXiv:2310.02244.
These are expository working notes. The paper should be taken as the source for precise statements, assumptions, and proofs.