Deep information propagation: from MLPs to residual networks

1. What is supposed to propagate?

Take two inputs, \(a\) and \(b\), and pass them through the same randomly initialized network. For a width-\(n\) hidden vector \(z^\ell\), track

The first measures signal size; the last measures alignment. For equal signal sizes \(q_a^\ell=q_b^\ell=q^\ell\), their squared distance per coordinate is

This identity explains why checking norms alone is insufficient. Two vectors can both have a perfectly healthy norm while becoming indistinguishable. Conversely, a growing distance can merely reflect exploding norms. We also need the input-to-hidden Jacobian, which measures how small perturbations and backward signals are amplified.

2. An MLP becomes a scalar variance recursion

Use preactivations as the state of a constant-width MLP:

Weights and biases are independent across layers. Conditional on the preceding features, each new coordinate is a centered Gaussian with variance

At large width, the average concentrates. If a typical coordinate of \(z^\ell\) has distribution \(\mathcal N(0,q^\ell)\), the next variance is therefore

A high-dimensional forward pass has become iteration of a scalar map. A fixed point \(q_*=V(q_*)\) gives a candidate persistent signal scale. Local attraction requires \(|V'(q_*)|<1\). The width factor \(1/n\) keeps one layer well scaled; the remaining constants decide what repeated composition does.

This is the mean-field reduction in Poole et al., Section 2. We take width to infinity at fixed depth before using the recursion at large depth. Gaussian weights make the conditional Gaussian step exact; replacing its empirical variance by a deterministic limit is the large-width step.

3. Why one input is not enough

Suppose both inputs have the same variance \(q\) and correlation \(c\). Their typical coordinates can be represented as

with independent standard Gaussians \(Z_1,Z_2\). Define the covariance produced by a fresh layer as

Using the same weights for both inputs makes the covariance depend on their old alignment. Using the same bias adds \(\sigma_b^2\) to their covariance as well as their variances. Thus

After the variance has settled to a positive \(q_*\), write this as \(c^{\ell+1}=C(c^\ell)\), with \(C(c)=K(q_*,c)/q_*\). If all initial correlations approach the same value, the network loses this measure of the original pairwise geometry. In particular, \(c\to1\) at fixed \(q_*\) implies \(d_\ell\to0\). The two-input construction is developed in Poole et al., Section 3.

4. The edge of chaos and the depth scale

Identical inputs remain identical, so \(C(1)=1\). The local question is whether nearly identical inputs move closer together or farther apart. Gaussian differentiation gives

and hence, at a positive variance fixed point,

Writing \(\delta_\ell=1-c^\ell\), the linearized recurrence is \(\delta_{\ell+1}\approx\chi\delta_\ell\). This gives the familiar classification:

In the ordered regime, \(\delta_\ell\approx\delta_0e^{-\ell/\xi_c}\), where

For example, \(\chi=0.9\) gives \(\xi_c\approx9.5\) layers, while \(\chi=0.99\) gives about 99.5. A small per-layer contraction matters after enough compositions. Near another attractive fixed point \(c_*\), the relevant derivative is \(C'(c_*)\), not necessarily \(C'(1)\). Variance relaxation has its own scale, \(\xi_q=-1/\log|V'(q_*)|\), when the derivative has magnitude strictly between zero and one.

Deep Information Propagation, Section 3, relates these depth scales to trainability. Their divergence removes an exponential obstruction; it does not promise that correlations stay unchanged forever or that optimization succeeds for every task.

5. Why the same factor controls gradients

The derivative of one MLP layer is \(J_\ell=W^{\ell+1}D_\ell\), where \(D_\ell=\operatorname{diag}(\phi'(z^\ell))\). A small forward perturbation is multiplied by \(J_\ell\); a backward signal is multiplied by its transpose. Under the usual mean-field independence approximation, a backward vector \(\delta^{\ell+1}\) obeys

When \(q^\ell\approx q_*\), this becomes \(g_0/g_L\approx\chi^L\). The same local sensitivity that contracts differences between inputs also contracts backward signals. These are squared norms; gradient RMS changes by approximately \(\chi^{L/2}\).

The original backward calculation explicitly uses a gradient-independence approximation: a loss gradient depends on the weights that produced it, so treating these objects as independent needs justification. See Section 4. This is an initialization diagnostic, not a recursion we can automatically reuse after feature learning creates new correlations.

6. A worked example: critical ReLU still forgets angles

Set \(\phi(z)=\max(0,z)\) and \(\sigma_b=0\). Half of a centered Gaussian’s second moment lies on its positive side, so

The He choice \(\sigma_w^2=2\) preserves \(q\) and gives \(\chi=1\). Every positive \(q\) is now a variance fixed point; there is no attraction to one preferred scale. But the normalized covariance still evolves. Evaluating the two-dimensional Gaussian integral gives

This map lies above \(c\) for \(-1\leq c<1\). Thus correlations increase despite critical initialization. To see the rate, put \(c=\cos\theta\). Then \(C(c)-c=(\sin\theta-\theta\cos\theta)/\pi\); expanding at small \(\theta\) yields

Consequently, \(\delta_{\ell+1}-\delta_\ell\approx-A\delta_\ell^{3/2}\), with \(A=2\sqrt2/(3\pi)\). Integrating this approximation gives \(\delta_\ell^{-1/2}\approx\delta_0^{-1/2}+A\ell/2\), and asymptotically \(1-c^\ell\sim9\pi^2/(2\ell^2)\). The forgetting is polynomial instead of exponential.

7. What criticality does not guarantee

Even perfect control of the average squared singular value of the full Jacobian \(J\) does not control its entire spectrum. For example, a matrix with half its singular values equal to \(\sqrt2\) and half equal to zero has mean squared singular value one, yet erases half the directions. Dynamical isometry asks for singular values concentrated near one, a much stronger condition. Pennington, Schoenholz, and Ganguli (2017) study this distinction and show why the activation and the weight ensemble both matter.

Finite width introduces another distinction: a stable mean need not describe a typical network well if fluctuations grow with depth. For ReLU MLPs, Hanin and Rolnick (2018) identify the sum of inverse layer widths as a control on activation-length fluctuations. At constant width this is \(L/n\). Taking width to infinity first suppresses an effect that can matter when depth grows alongside width.

These observations suggest a hierarchy of checks: second moments, input geometry, the spread of Jacobian gains, and variation across random seeds. Passing one does not eliminate the need to inspect the others.

8. Residual connections change the calculation

For a residual network, use \(x^\ell\) for the stream and write

The old state is carried forward explicitly. To isolate the mechanism, consider the branch

with the same independent Gaussian initialization as above and a centered Gaussian initial embedding \(x^0\). This choice ends the branch with a fresh centered affine map. Conditional on the incoming stream, \(\mathbb E[F_\ell(x^\ell)\mid x^\ell]=0\). It is a convenient model for deriving moment recursions, rather than a literal Transformer block.

Expanding the squared norm makes the role of this centering precise:

The cross term vanishes in expectation at initialization under the conditional centering above. Thus the leading variance increment is quadratic in the multiplier. This conclusion uses both the architecture and the randomness; it is not an algebraic consequence of having a skip connection.

9. Adding covariance instead of replacing it

Apply the preceding expansion to both a single stream and a pair of streams. In the wide Gaussian limit of this model, the functions \(V\) and \(K\) defined above give

An MLP replaces its old covariance by the transformed covariance. A residual network adds a small transformed covariance to what was already there. In particular, positive branch variance generally makes \(q^\ell\) grow; we should not impose the MLP’s fixed-point equation \(q_*=V(q_*)\) on this residual stream.

For equal-norm inputs, dividing the two recursions gives

There are two ways for this change to be small: use a small branch multiplier, or have the residual stream large relative to the new branch. The second can slow angular changes while allowing norms to grow. This is another reason to keep \(q\) and \(c\) separate.

For a deeper branch such as \(W_2\phi(W_1x)\), the same covariance-addition argument applies at initialization when the final random matrix supplies conditional centering. Its internal covariance calculation changes \(V\) and \(K\). Attention and normalization require their own corresponding calculation.

10. The sum of squared residual multipliers

For the bias-free ReLU branch above, put \(\kappa=\sigma_w^2/2\). Then \(V(q)=\kappa q\), so the variance recursion is exactly solvable:

For small multipliers, this logarithm is approximately \(\kappa\sum_\ell a_\ell^2\). More generally it is bounded above by that quantity. Define

For a uniform multiplier \(a_\ell=A L^{-\alpha}\), we have \(S_L=A^2L^{1-2\alpha}\). The consequences are visible without any training calculation:

Bounded \(S_L\) is the central scaling condition in Stable ResNet, under that paper’s assumptions. It also accommodates nonuniform, square-summable branch multipliers. This is the variance-level version of the \(\sqrt L\,L^{-\alpha}\) argument in the TP VI note.

“Finite” is an asymptotic statement: a very large fixed \(A\) can still give an unusably large \(e^{\kappa A^2}\). Choosing a depth exponent removes systematic growth with model size; the remaining multiplier is still a hyperparameter.

11. A worked residual example: slowing the correlation map

For the same ReLU model, \(K(q,c)=\kappa q C_{\mathrm{ReLU}}(c)\). Therefore

The MLP takes a full step through its correlation map at every layer. This residual network takes a fraction \(\beta_\ell\) of that step. Correlation still increases, but more slowly. With \(\kappa=1\), \(L=1000\), and \(c^0=0.5\), direct iteration gives:

For small steps, introduce an accumulated coordinate \(\tau_\ell=\sum_{j<\ell}\beta_j\). The recurrence is then an Euler approximation to

With \(a=A/\sqrt L\), increasing \(L\) refines the discretization of a finite interval \(\tau_L\approx\kappa A^2\). It does not keep running this correlation evolution for longer and longer. With \(a=A/L\), the initialization interval itself shrinks to zero. This explains the table: the latter stack initially preserves its input geometry almost unchanged.

Likewise, the unscaled row’s correlation approaching one does not imply that its absolute distance vanishes: \(d_L^2=2q^L(1-c^L)\) also contains an exploding \(q^L\). “Becoming parallel” and “becoming close” coincide only when the norm scale is controlled.

12. Identity paths and gradient propagation

A residual block has Jacobian

For a perturbation \(v\), expansion gives \(\|J_\ell v\|^2=\|v\|^2+2a_\ell v^\top B_\ell v+a_\ell^2\|B_\ell v\|^2\). In the centered isotropic initialization calculation, the cross term averages to zero. If the branch’s mean-square gain is \(\chi_{F,\ell}\), the block gain is consequently

The backward expression again uses the mean-field independence approximation. For our ReLU example, \(\chi_{F,\ell}=\kappa\), giving the same product as the forward variance. The identity path supplies the one; residual sensitivity still accumulates. An unscaled branch with He variance has gain two per block in this model, not one.

This also clarifies why residual criticality is not obtained by simply imposing the MLP equation \(\chi=1\) on each branch. The relevant product contains \(1+a^2\chi_F\). If branch gains stay bounded, bounded \(S_L\) keeps this average product bounded. It does not guarantee every singular value is well conditioned.

Activation choice matters. In our model a bounded activation satisfies \(V(q)\leq\sigma_w^2\|\phi\|_\infty^2+\sigma_b^2\), so even unscaled variance grows at most linearly; ReLU instead makes \(V(q)\) proportional to \(q\). Yang and Schoenholz show activation-dependent polynomial and subexponential behavior in their residual architectures, with exponential gradient growth still possible for ReLU. “Skip connections put every network safely at the edge of chaos” is too broad.

13. Centering, normalization, and the location of the skip

A nonzero mean changes the power counting

If a branch contributes a persistent mean \(m\mathbf1\), that contribution accumulates like \(m\sum_\ell a_\ell\). With \(a=A/\sqrt L\), this is \(Am\sqrt L\), even though \(S_L\) is bounded. The quadratic variance argument controls centered random increments; it cannot control an order-one coherent mean. This is precisely why TP VI centers its post-activation branches. After training, even zero coordinate mean does not rule out correlation with the stream or coherent learned directions.

Pre-normalization does not normalize the residual stream itself

In \(x^{\ell+1}=x^\ell+a_\ell F_\ell(\operatorname{LN}(x^\ell))\), LayerNorm regulates what the branch sees. The stream still adds a new term. If the centered branch has approximately constant variance \(v\), its initialization recursion is \(q^L\approx q^0+vS_L\), so unscaled branches can still make the stream grow with depth. Normalization also changes the branch Jacobian; the unnormalized ReLU formulas cannot simply be reused.

The identity must remain an identity

Applying an activation or normalization after the residual addition gives a Jacobian of the form \(D\psi\,(I+aB)\), so the direct route is multiplied by \(D\psi\) too. This placement issue motivates the identity-path analysis of He et al. (2016). For Transformers, Xiong et al. (2020) analyze how pre-LN and post-LN affect initialization gradients. Attention adds token-to-token covariance as well; a two-input scalar recursion is an explanatory model, not a complete Transformer theory.

14. Connecting initialization to Tensor Programs IV–VI

The connection to TP IV and V starts with the same law-of-large-numbers substitution: an empirical average over neurons becomes an expectation of a representative coordinate. Information propagation uses it for random initialization. Tensor Programs supplies machinery for tracking the richer dependencies that arise when training reuses weights and aligns updates with features. A stable initial covariance kernel does not determine whether that kernel subsequently learns.

There is a useful parallel with the residual calculations. At initialization, centering removed a cross term and left a contribution proportional to \(a^2\). Training produces correlated updates, whose direct residual effect is proportional to \(a\Delta\theta\). With width factors suppressed,

Bounded random accumulation gives \(\alpha\geq1/2\). Stable, nonvanishing first-order learning motivates \(\alpha+\gamma=1\). These are the two balances in TP VI. Initialization selects a range of possible scalings; an additional learning criterion selects a point within it.

Depth-\(\mup\) chooses \(\alpha=\gamma=1/2\) to retain nonvanishing random variation across depth. That is stronger than requiring good signal propagation. The residual table above shows that \(1/L\) can preserve initial geometry even more closely; it simply performs less random transformation before learning.

15. Why CompleteP can prefer a smooth depth limit

CompleteP asks that learning continue to use nonlinear interactions inside a block. For a two-weight branch, expanding its change produces terms linear in \(\Delta W_1,\Delta W_2\) and a mixed term containing \(\Delta W_2\Delta W_1\). Along the first-order balance \(\Delta\theta=O(\eta L^{\alpha-1})\), their characteristic residual scales are

assuming nonzero generic terms. For \(\alpha<1\), depth suppresses the mixed interaction relative to the linear response. At \(\alpha=1\), this extra suppression disappears. The CompleteP criterion thus favors \(a=1/L\) with depth-independent internal learning.

There is no contradiction with information propagation. At initialization, \(S_L=1/L\to0\), so the residual stack approaches the identity on its initial embedding. During training, \(L\) learned increments of size \(1/L\) can build a finite transformation. With fractional depth \(\lambda=\ell/L\), bounded learned branches suggest a smooth evolution in \(\lambda\), while their dependence on trainable parameters remains nonlinear.

This resembles the small initial readout in μP: a random contribution can vanish in the size limit while a correlated learned contribution survives. Demanding maximal random activity at initialization would miss that possibility.

These different selection criteria motivate treating initial input geometry, variation across depth, and nonlinear learning within blocks as distinct design questions. The next section explains what each viewpoint contributes to the others.

16. How the theories fit together

The higher-level problem is to make a larger network remain a useful learning system. That requires both a representation through which signals can travel and parameter updates that can improve it. Deep information propagation and the parametrization papers examine different ways this can fail. Their scopes overlap: TP IV–VI and CompleteP already include initialization stability, and the Tensor Programs machinery can also describe initialization statistics. The value of the information-propagation viewpoint is the explicit diagnosis of signal geometry and sensitivity, beyond merely checking their order of magnitude.

What signal propagation adds to a scaling prescription

A width exponent does not fix a network’s effective usable depth. For example, \(W_{ij}\sim\mathcal N(0,\sigma_w^2/n)\) has the same width scaling for every fixed \(\sigma_w\), yet changing this constant can change the correlation map and the gradient multiplier \(\chi\). An MLP can be well scaled as \(n\to\infty\) at fixed \(L\), while its sensitivity is multiplied by \(\chi^L\) and becomes extremely small at the depth of interest. DIP’s depth scales help expose this distinction. For residual networks, the analogous example is \(a=A/\sqrt L\): the depth exponent controls accumulation, but the finite factor \(e^{\kappa A^2}\) in our worked example can still be very large.

Similarly, finite feature norms and nonvanishing feature updates do not by themselves ensure that two inputs remain distinguishable. CompleteP’s preservation of nonlinear parameter interactions does not by itself establish that those interactions produce useful geometry for a task. The \(q,c,d\) diagnostics provide another check. They can help identify poor initialization constants or architectural choices within a scaling family, although they do not determine the best trained representation. Control of the full Jacobian spectrum requires the additional dynamical-isometry analysis, beyond DIP’s scalar mean-field factors.

What the learning theories add to signal propagation

Conversely, the very same initialized network can be trained with learning rates that freeze its hidden features, make them change at order one, or destabilize them as width grows. Its initial \(q\), \(c\), and Jacobian cannot distinguish these choices. TP IV classifies feature learning under width scaling; TP V connects the resulting parametrization to tuning on smaller models. These contributions address something that good initial propagation alone cannot select: how parameter updates should scale, including the distinct roles of input, hidden, and output layers.

The same gap appears in depth. A bounded sum \(\sum_\ell a_\ell^2\) controls centered random accumulation at initialization, but does not prescribe the coherent learned sum \(\sum_\ell a_\ell\Delta\theta^\ell\). TP VI analyzes this training balance and selects a limit using feature diversity. CompleteP examines the further possibility that features move while multi-operation blocks become linearized in their own parameter updates. Initialization statistics cannot detect that loss of nonlinear learning.

How to use the contributions together

A useful workflow is to choose an architecture and a candidate width-and-depth parametrization together, examine signal propagation at the intended depths, and then check the scale and nonlinear character of actual training updates. Signal propagation helps assess the initialization constants and geometry; the learning theories constrain how the optimizer and forward multipliers should change with model size; transfer experiments test whether those choices remain useful. A failure at any stage can motivate revisiting the earlier choices.

This is not a hierarchy in which CompleteP satisfies every criterion of TP VI. Within the scaling family considered here, preserving maximal random variation across depth and preserving nonlinear interactions inside multi-layer blocks favor different exponents. Nor should the MLP condition \(\chi=1\) be imposed indiscriminately on residual branches, whose Jacobians contain an identity term. The combined perspective makes these choices explicit; it does not supply a universal theorem of optimal hyperparameters, fast optimization, or generalization for every finite network.

17. What to measure in an actual network

The derivations suggest a compact diagnostic sequence, using several depths and random seeds:

1. Measure both the stream and the branch. Track coordinate means and RMS values. For a residual network, also measure \(a_\ell F_\ell(x^\ell)\) and its inner product with the stream; a large cross term invalidates a variance-only explanation.
2. Propagate pairs of inputs. Track \(q_a,q_b,c_{ab}\) and the actual distance. Use nearby and unrelated pairs. This distinguishes shrinking norms, increasing alignment, and excessive local sensitivity.
3. Probe the Jacobian. Compare the gains of several random tangent or backward vectors. For manageable models, inspect singular values too. An average gain near one can hide badly distorted directions.
4. Then take training steps. Check feature changes, each block’s direct contribution, and the discrepancy from linearizing a multi-layer block in its parameters. These are the additional checks needed to connect initialization to μP and CompleteP.

Stochastic layers need an explicit convention. The dropout result in Deep Information Propagation uses independent masks for the two inputs: even identical inputs then receive different perturbations, and the \(c=1\) fixed point is lost. A correlation test with shared masks asks a different question. Likewise, a controlled mean-field initialization does not by itself establish training speed, generalization, or hyperparameter transfer.

18. Five equations to remember

The last sum is schematic coherent power counting, with a nonvanishing learning effect desired. The central distinction is between propagating an existing signal and learning a new transformation of it. Residual connections make these tasks separable: the identity path carries the signal, while the scaled branch controls how much random or learned change is added.

References

1. Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. NeurIPS 2016. Paper. Mean-field variance and correlation maps.
2. Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. ICLR 2017. Paper. Depth scales, backward propagation, and dropout.
3. Greg Yang and Samuel S. Schoenholz. Mean Field Residual Networks: On the Edge of Chaos. NeurIPS 2017. Paper. Activation-dependent residual dynamics.
4. Soufiane Hayou, Eugenio Clerico, Bobby He, George Deligiannidis, Arnaud Doucet, and Judith Rousseau. Stable ResNet. AISTATS 2021. Paper. Residual scaling and bounded sums of squared multipliers.
5. Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NeurIPS 2017. Paper.
6. Boris Hanin and David Rolnick. How to Start Training: The Effect of Initialization and Architecture. NeurIPS 2018. Paper. Finite-width fluctuations and residual architectures.
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. ECCV 2016. Paper.
8. Ruibin Xiong et al. On Layer Normalization in the Transformer Architecture. ICML 2020. Paper.
9. Greg Yang et al. Tensor Programs IV, V, and VI. IV · V · VI.
10. Nolan Dey et al. Don’t be lazy: CompleteP enables compute-efficient deep transformers. NeurIPS 2025, corrected January 2026 version. Paper.

These are expository working notes. Numerical tables are iterations of the displayed infinite-width ReLU recursions, not measurements of trained networks. The residual worked example fixes the activation order and centering explicitly; its exact formulas should not be transferred unchanged to other architectures.

Related notes: Tensor Programs IV & V · Tensor Programs VI · CompleteP.