The cleanest way I have found to understand CompleteP is to start from the block-depth limitation at the end of the TP VI note. A deep residual network can have stable activations and order-one feature movement while the trainable layers inside each block lose their nonlinear interactions. CompleteP asks that those interactions survive too. This changes which depth scaling we should prefer.
These notes follow Dey et al., Don’t be lazy: CompleteP enables compute-efficient deep transformers, using the corrected January 2026 version. The derivations below unpack its scaling arguments; they are not a general convergence proof for Transformer training.
1. The big picture
The TP IV & V note asks how to preserve feature learning as width grows. The TP VI note asks the corresponding question as the number of residual blocks grows. CompleteP adds a more local test: when a block contains several trainable operations, does training still use their nonlinear composition?
The paper organizes the problem around three requirements:
- Stable initialization: features and outputs must not blow up with model size.
- Maximal residual stream updates: training must produce a finite, nonvanishing total feature change, with every block contributing at the appropriate scale.
- Complete feature learning: nonlinear dependence on trainable parameters must not disappear merely because the network becomes wider or deeper.
The first two leave a family of residual scalings \(L^{-\alpha}\), with \(1/2\leq\alpha\leq1\). The third singles out \(\alpha=1\) within this family. See the paper’s Section 6.
2. Separating width, depth, and block depth
Write a residual network as
\[ h^{\ell+1}=h^\ell+L^{-\alpha}F_\ell(h^\ell;\theta^\ell), \qquad h^\ell\in\mathbb R^n. \]There are three different sizes here. The width is \(n\). The number of residual blocks is \(L\). The block depth is the number of composed trainable operations inside \(F_\ell\), which stays fixed as \(L\) increases. A Transformer MLP branch such as
\[ F_\ell(h)=W_2^\ell\phi\!\left(W_1^\ell\operatorname{LN}(h)\right) \]already has block depth two.
First suppress the width factors and analyze powers of \(L\). Write the characteristic internal update scale as
\[ \Delta\theta^\ell=O(L^{-\gamma}). \]This is a depth-scaling shorthand in width-normalized coordinates. It does not say that every entry of a wide matrix moves by order one. We restore the entrywise width factors in Section 11.
3. What initialization stability actually forces
At initialization, assume centered residual outputs with order-one coordinate variance and fresh randomness across blocks. The random contributions accumulate on the scale
\[ \sqrt L\,L^{-\alpha}=L^{1/2-\alpha}. \]Equivalently, their accumulated variance is of order \(L^{1-2\alpha}\). Stability therefore requires
\[ \boxed{\alpha\geq\frac12.} \]At \(\alpha=1/2\), the residual randomness survives at order one. At \(\alpha=1\), it vanishes like \(L^{-1/2}\). But the identity path still carries the order-one input representation, so the residual stream itself need not vanish.
This is the first distinction from the TP VI selection principle: stable initialization does not require a nonvanishing random contribution from the entire residual stack. CompleteP accepts an initially near-identity stack so that its blocks can subsequently learn strongly.
The centering assumption matters. An order-one nonzero mean can accumulate like \(L\), so a variance calculation alone cannot justify arbitrary residual architectures. The paper’s Appendix C.1 gives the signal-propagation calculation for its setup.
4. What a maximal residual update forces
The direct first-order change from updating one block is schematically
\[ \Delta_{\theta^\ell}h^{\ell+1} \approx L^{-\alpha}D_\theta F_\ell\,\Delta\theta^\ell =O(L^{-\alpha-\gamma}). \]Training updates are correlated through the data and the backward signal. Their learned contributions can accumulate coherently across depth, giving
\[ L\cdot L^{-\alpha-\gamma}=L^{1-\alpha-\gamma}. \]The critical balance is therefore the same one used in the TP VI note:
\[ \boxed{\alpha+\gamma=1,\qquad \Delta\theta^\ell=O(L^{\alpha-1}).} \]Every block contributes an order-\(1/L\) feature increment, so \(L\) blocks can jointly produce an order-one change. Requiring bounded internal updates gives \(\gamma\geq0\), hence \(\alpha\leq1\). Together with initialization stability, this leaves
\[ \frac12\leq\alpha\leq1. \]5. Why moving features can still be lazy
Fix a block’s input \(h\) and expand its output around its initial parameters:
\[ F(h;\theta_0+\Delta\theta) =F(h;\theta_0)+D_\theta F\,\Delta\theta +\frac12D_\theta^2F[\Delta\theta,\Delta\theta]+\cdots. \]For \(\alpha<1\), the depth factor in \(\Delta\theta\) tends to zero. Each extra power of the update then introduces another vanishing factor. A block can change through its first derivative while its higher-order parameter interactions become negligible.
That does not imply that the entire residual network is a fixed kernel model. Its input to each block may still change substantially through upstream training. The defect is more specific: conditional on a block’s input, learning inside that block approaches its parameter linearization.
For \(W_2\phi(W_1h)\), we want the second matrix to learn how to use features that the first matrix is itself learning. A first-order approximation retains separate changes through \(W_1\) and \(W_2\), but drops the interaction between their learned changes. The next calculation isolates exactly that loss.
6. The two-weight calculation
Use the paper’s scalar toy block, with width one:
\[ h^+=h+L^{-\alpha}w_2w_1h. \]Although it is linear in \(h\), it is nonlinear in the pair \((w_1,w_2)\). Hold \(h\) fixed and update both weights. The exact output change is
\[ \Delta h^+ =\underbrace{L^{-\alpha}h(w_2\Delta w_1+w_1\Delta w_2)}_{T_1} +\underbrace{L^{-\alpha}h\,\Delta w_2\Delta w_1}_{T_2}. \]Here \(T_1\) is the linearized update, and \(T_2\) is the interaction that linearization discards. With order-one \(h,w_1,w_2\) and generic, non-cancelling updates of size \(\eta L^{\alpha-1}\),
\[ T_1=\Theta(\eta L^{-1}),\qquad T_2=\Theta(\eta^2L^{\alpha-2}). \]Thus the useful diagnostic is a relative error:
\[ \boxed{\frac{|T_2|}{|T_1|}=\Theta(\eta L^{\alpha-1}).} \]| Depth exponent | Nonlinear / linear contribution | As depth grows |
|---|---|---|
| \(\alpha=1/2\) | \(\Theta(\eta/\sqrt L)\) | vanishes |
| \(\alpha=3/4\) | \(\Theta(\eta/L^{1/4})\) | vanishes, more slowly |
| \(\alpha=1\) | \(\Theta(\eta)\) | retains its depth scale |
At CompleteP, both terms carry the same \(1/L\) factor. The quadratic term is still smaller by a factor of the learning rate when \(\eta\) is small, but increasing depth does not suppress it further.
This is the calculation in Section 6, equations (5)–(6), with the fixed base learning rate made explicit.
7. Why all nonlinear orders select \(\alpha=1\)
The same depth counting applies to a generic nonzero \(k\)-th order term in a fixed-depth block. Suppressing derivative constants and assuming the width normalization has already been handled,
\[ T_k=O\!\left(\eta^kL^{-\alpha+k(\alpha-1)}\right). \]Relative to the first-order scale \(\eta/L\),
\[ \frac{T_k}{T_1} =O\!\left(\eta^{k-1}L^{(k-1)(\alpha-1)}\right). \]For every \(k>1\), \(\alpha<1\) introduces an additional depth-dependent suppression. At \(\alpha=1\), each surviving Taylor order has the same depth factor:
\[ \boxed{T_k=O(\eta^k/L).} \]“All orders survive” means that depth no longer forces them away. A particular derivative can still be zero, a small learning rate still suppresses high orders, and this power counting by itself does not prove convergence of an infinite Taylor series.
Combining the three requirements now selects
\[ \boxed{\alpha=1,\qquad\gamma=0.} \]This is the core CompleteP depth prescription: scale the branch by \(1/L\) and keep its internal feature-learning scale independent of depth.
8. What “complete” means
For a representation \(H\) and a selected parameter subset \(S\), compare the actual change from updating only \(S\) with the change predicted by linearization at initialization. Schematically, define
\[ R_S= \frac{\|\Delta_S H-\Delta_S H^{\mathrm{lin}}\|} {\|\Delta_S H^{\mathrm{lin}}\|}. \]Where the linear response is nonzero, \(R_S\to0\) detects laziness with respect to \(S\), provided \(H\) was nonlinear in that subset at finite size. Complete feature learning asks that no such nonlinear dependence be lost asymptotically for the relevant hidden representations and output.
The subset language matters. In the toy block, the map is already linear in \(w_1\) alone when \(w_2\) is fixed; it makes no sense to demand curvature there. The pair \(\{w_1,w_2\}\) is the subset with a nonlinear interaction to preserve.
The uniqueness claim concerns the paper’s scaling family and assumptions. It is not a claim that one literal implementation is the only possible parametrization for every architecture.
9. Reconciling CompleteP with Tensor Programs VI
The apparent conflict is real and instructive. TP VI prefers \(1/\sqrt L\) because it preserves random variation across neighboring depths. CompleteP prefers \(1/L\) because it preserves nonlinear learning inside multi-layer blocks. These are different properties.
| Property | Depth-\(\mup\) | CompleteP |
|---|---|---|
| Residual multiplier | \(L^{-1/2}\) | \(L^{-1}\) |
| Internal update’s depth factor | \(L^{-1/2}\) | \(1\) |
| Total initial residual randomness | \(O(1)\) | \(O(L^{-1/2})\) |
| Direct learned effect per block | \(O(1/L)\) | \(O(1/L)\) |
| Quadratic / linear update scale | \(O(\eta/\sqrt L)\) | \(O(\eta)\) |
| Selection principle | maximal feature diversity across depth | complete nonlinear feature learning |
The clean TP VI optimality result concerns its one-matrix residual blocks. Its multi-layer-block limitation is precisely where the two-weight calculation becomes relevant. CompleteP revisits which criterion should guide practical Transformer scaling.
An ODE-like trajectory in the depth coordinate can still depend nonlinearly on trainable parameters. Smoothness as a function of \(\ell/L\) does not imply linearity as a function of \(\theta\). Likewise, an initially near-identity stack need not stay near identity after training: its \(L\) learned order-\(1/L\) contributions can sum to order one.
How to read the earlier note. TP VI’s objection to the smooth depth limit belongs to its feature-diversity criterion. It should not be read as a universal proof that \(1/L\) scaling cannot transfer hyperparameters or train deep Transformers effectively. CompleteP supplies a different criterion and empirical evidence for that regime.
CompleteP selects \(1/L\) to preserve nonlinear learning inside each block. With branch multiplier \(L^{-\alpha}\), maximal residual updates require an internal update scale \(\Delta\theta=O(\eta L^{\alpha-1})\), suppressing width factors. For \(\alpha<1\), these updates shrink with depth: in a two-weight block, the quadratic interaction between their updates becomes smaller than the linear response by a factor \(O(\eta L^{\alpha-1})\). At \(\alpha=1\), internal learning retains its depth scale, while the outer \(1/L\) multiplier keeps each block’s residual contribution small. The choice follows from preserving nonlinear parameter interactions; smoothness is a consequence. See Section 6.
In this interpretation, smoothness concerns the fractional depth \(\lambda=\ell/L\): bounded branches give \(O(1/L)\) increments, and \(L\) such increments can build an \(O(1)\) transformation. Small steps through depth can therefore implement strongly nonlinear learned computations. Preserving the balance between linear and nonlinear learning as depth changes is the proposed reason that hyperparameters tuned on a shallow model remain useful in a deeper one. The paper’s transfer experiments and compute-efficiency results support this prescription for its Transformer setup.
10. SGD versus Adam: scale the update first
The invariant requirement is
\[ \Delta\theta^\ell=O(L^{\alpha-1}), \]after factoring out the width scaling. The learning rate needed to realize it depends on the optimizer and parameter coordinates.
Adam or AdamW
The raw gradient includes the branch multiplier \(L^{-\alpha}\). Adam’s ratio of a first moment to a root second moment cancels this common gradient scale, when epsilon is scaled consistently or is negligible. The learning rate must therefore provide
\[ \eta_{\mathrm{Adam,block}}\propto L^{\alpha-1}. \]For CompleteP, this has no additional depth factor. Width factors still apply.
SGD
In the same direct-weight coordinates, SGD retains the \(L^{-\alpha}\) gradient factor. Thus
\[ \eta_{\mathrm{SGD,block}}L^{-\alpha} \propto L^{\alpha-1} \quad\Longrightarrow\quad \eta_{\mathrm{SGD,block}}\propto L^{2\alpha-1}. \]At \(\alpha=1/2\), its extra depth factor is one. At \(\alpha=1\), it is \(L\). This explains why copying an Adam learning-rate rule into SGD changes the intended dynamics. These are depth factors on top of the appropriate widthwise rules, not universal raw learning rates for all parameters.
11. Putting the width factors back
For an \(n\times n\) hidden matrix in the usual direct-weight \(\mup\) coordinates, initialization has entry scale \(n^{-1/2}\). A learned update acts coherently on a correlated width-\(n\) feature vector. Consequently,
\[ \Delta W_{ij}=O\!\left(\frac{L^{\alpha-1}}n\right) \quad\Longrightarrow\quad (\Delta Wx)_i=O(L^{\alpha-1}). \]This is the same random-versus-learned summation distinction used in the IV & V note. Applying the residual multiplier gives
\[ L^{-\alpha}(\Delta Wx)_i=O(1/L). \]At CompleteP, therefore,
\[ W_{ij,0}=O(n^{-1/2}),\qquad \Delta W_{ij}=O(n^{-1}),\qquad (\Delta Wx)_i=O(1). \]The matrix update becomes small entrywise with width, yet its effect on learned features stays finite. This is why entrywise motion alone cannot diagnose laziness: one must include the correlated sums in the network’s actual response.
CompleteP retains the input / hidden / output distinction of widthwise \(\mup\); the calculation above concerns hidden matrices only. The boundary layers are the global embedding and readout; matrices inside residual blocks follow the hidden-layer rules. Compared with the IV & V note, the main change here is from SGD to AdamW: gradient normalization changes the learning rates needed to obtain the same widthwise update scales. The explicit inverse-width readout multiplier is a different way of storing the same effective \(O(n^{-1})\) output weights. These differences preserve the underlying widthwise \(\mup\) principle. CompleteP’s substantive extension is in depth: the \(1/L\) residual multiplier and the requirement that nonlinear learning inside each block survive as depth grows.
12. The practical AdamW scaling table
Implement relative to a base model:
\[ m_n=\frac{n}{n_0},\qquad m_L=\frac{L}{L_0}. \]The following multipliers follow Table 1 of the corrected paper. Multiply each base value by the listed factor. “Hidden” means linear matrices in the Transformer backbone. The readout uses an explicit forward multiplier; its stored initialization and LR therefore differ from the effective-readout SGD convention in the IV & V note.
| Quantity | General \(\alpha\) | CompleteP |
|---|---|---|
| Attention and MLP branch output | \(m_L^{-\alpha}\) | \(m_L^{-1}\) |
| Hidden weight initialization variance | \(m_n^{-1}\) | \(m_n^{-1}\) |
| Hidden weight AdamW LR | \(m_n^{-1}m_L^{\alpha-1}\) | \(m_n^{-1}\) |
| Hidden bias AdamW LR | \(m_L^{\alpha-1}\) | \(1\) |
| Pre-LN gain and bias AdamW LR | \(m_L^{\alpha-1}\) | \(1\) |
| Hidden weight decay coefficient | \(m_n\) | \(m_n\) |
| Embedding and stored readout initialization variance | \(1\) | \(1\) |
| Embedding, readout, and final-LN AdamW LR | \(1\) | \(1\) |
| Readout forward multiplier | \(m_n^{-1}\) | \(m_n^{-1}\) |
| Adam epsilon inside residual blocks | \(m_n^{-1}m_L^{-\alpha}\) | \(m_n^{-1}m_L^{-1}\) |
| Adam epsilon for embedding and readout | \(m_n^{-1}\) | \(m_n^{-1}\) |
Hidden weight standard deviation thus scales as \(m_n^{-1/2}\), with no extra depth shrinkage. LN initial values keep their base scale. For the paper’s untied embedding/readout convention, the logits have the form
\[ f=m_n^{-1}W_{\mathrm{out}}\operatorname{LN}_{\mathrm{final}}(h^L). \]Both attention and MLP branch outputs receive the depth multiplier before their residual additions. Embeddings, the final normalization, and the readout are not repeated \(L\) times, so they do not receive that branch factor.
13. Why biases and LayerNorm have their own rules
A matrix update gains a coherent factor of \(n\) when it acts on a feature vector. A bias update does not: it is added directly to a coordinate. To obtain the same internal feature scale, the required updates are
\[ \Delta W_{ij}=O(n^{-1}L^{\alpha-1}),\qquad \Delta b_i=O(L^{\alpha-1}). \]A LayerNorm gain or shift also acts coordinatewise, so it follows the bias rule rather than the hidden-matrix rule. This accounts for the absence of an inverse-width LR multiplier on those parameters.
It also matters for comparisons with \(\alpha=1/2\). Shrinking matrix learning rates with depth while leaving pre-LN and bias learning rates fixed lets those parameters move too strongly. The paper’s corrected \(\alpha=1/2\) baseline scales these groups too; its comparison with CompleteP is not just a comparison against that incomplete recipe.
At \(\alpha=1\), these depth factors all become one. The derivations are in Appendix D.
14. Weight decay and Adam’s epsilon
Weight decay must produce the intended feature change
AdamW includes a separate shrinkage update,
\[ \Delta W_{\mathrm{decay}}=-\eta_W\lambda_W W. \]Since \(Wx\) is order one, its effect on an internal feature is of scale \(\eta_W\lambda_W\). With
\[ \eta_W=\eta_0m_n^{-1}m_L^{\alpha-1},\qquad \lambda_W=\lambda_0m_n, \]this becomes \(\eta_0\lambda_0m_L^{\alpha-1}\), matching the depth scale of the learned internal update. There is no additional depth multiplier on \(\lambda_W\). At CompleteP, the shrinkage fraction \(\eta_W\lambda_W\) is independent of both width and depth.
This argument concerns hidden matrices to which decay is applied. It does not prescribe turning on decay for biases or normalization parameters that the base recipe excludes.
Epsilon must follow the raw gradient scale
Ignoring the separate decay term, Adam’s update is
\[ \Delta\theta=-\eta\frac{\widehat m}{\sqrt{\widehat v}+\epsilon}. \]If gradients scale by a positive factor \(c\), then \(\widehat m\) and \(\sqrt{\widehat v}\) both scale by \(c\). Their ratio is invariant only if epsilon is negligible or scales by \(c\) too. For residual-block parameters in this convention, the characteristic gradient factor is
\[ c=m_n^{-1}m_L^{-\alpha}. \]Hence the epsilon row in the table. Otherwise, as the model grows, a formerly small fixed epsilon can dominate the denominator and silently change the update scaling. Outside the residual stack, the embedding/readout gradient lacks the depth factor. See Appendix D.5.
15. What actually gets transferred
As in \(\mu\)Transfer, copy the base hyperparameters and regenerate the model-size-dependent values. For example, suppose the target is twice as wide and four times as deep as the base model:
\[ m_n=2,\qquad m_L=4. \]CompleteP gives the following changes relative to that base model:
| Quantity | Target / base |
|---|---|
| Residual branch multiplier | \(1/4\) |
| Hidden weight standard deviation | \(1/\sqrt2\) |
| Hidden weight AdamW LR | \(1/2\) |
| Bias, pre-LN, embedding, and readout LR | \(1\) |
| Hidden weight decay coefficient | \(2\) |
| Readout multiplier | \(1/2\) |
| Residual-block Adam epsilon | \(1/8\) |
| Embedding/readout Adam epsilon | \(1/2\) |
The purpose is to preserve what a base LR or initialization scale does to training. CompleteP also preserves the balance between linear and nonlinear internal learning terms, so the finite proxy and the deeper target need not drift toward different local learning regimes.
This is a mechanism for expecting transfer, not a theorem that the exact optimum is identical at every finite size. Changing batch size, training duration, data, or architecture introduces additional questions beyond the width-and-depth exponents.
16. What the experiments establish
The paper tests pre-LN decoder-only language models on SlimPajama, comparing standard parametrization, width-only \(\mup\), and the corrected depth scalings \(\alpha=1/2\) and \(\alpha=1\). In its fixed-token experiments, CompleteP gives more stable optima for base learning rate and initialization scale as depth changes. The compute-oriented experiments also test models trained at 20 tokens per parameter. See Section 4.
For models with roughly 1.5 billion non-embedding parameters, the paper reports about 11.8% estimated FLOP savings over its \(\mup\) baseline at the best tested shape, and 34.4% for the deepest, 179-layer shape. These are equal-loss comparisons using fitted scaling curves, not universal training-speed guarantees. The broader finding is that a larger range of narrow, deep models remains competitive. See Section 5.
The conceptual lesson is that an apparent architectural disadvantage can partly be a parametrization disadvantage. Before concluding that depth is inefficient, the deeper models need a scaling rule that lets their blocks learn on comparable terms.
17. How to check an implementation
A useful check follows the same order as the derivation:
- Check initialization. Across widths and depths, inspect residual-stream RMS values and logits. They should remain controlled.
- Check early training. Run a few steps on the same data with fixed base HPs. Inspect feature changes, including bias and normalization contributions. An individual block’s direct residual effect should carry the inverse-depth scale.
- Check nonlinear response. Freeze the input to a selected multi-layer block and compare its actual parameter-update response with a Jacobian-vector linearization. Under CompleteP, the relative discrepancy should not systematically disappear just because depth grows.
- Check transfer empirically. Sweep the same base LR and initialization coordinates on a few sizes. Compare both the location of the optimum and the achieved loss.
Stable coordinate checks verify a necessary scaling property; they do not establish complete feature learning or successful HP transfer by themselves. The authors provide a minimal implementation and coordinate-check example for the pre-LN setup.
18. Five equations to remember
The small residual contribution and the full internal learning scale coexist. Widthwise \(\mup\) handles the correlated sums inside a block; CompleteP’s inverse-depth multiplier handles the sum over blocks. This lets depth grow without automatically turning each block’s nonlinear learning into its tangent approximation.
References
- Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestness. Don’t be lazy: CompleteP enables compute-efficient deep transformers. NeurIPS 2025. arXiv:2505.01618v4 (19 January 2026). Especially Section 6, Table 1, and Appendices C–E.
- Greg Yang and Edward J. Hu. Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks. ICML 2021. PMLR.
- Greg Yang et al. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. 2022. arXiv:2203.03466.
- Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks. ICLR 2024. arXiv:2310.02244.
- EleutherAI and contributors. nanoGPT-mup, CompleteP branch. Reference implementation.
These are expository working notes. The original papers are the source for precise assumptions, definitions, and claims. The two-weight example and scaling prescriptions are explained from Dey et al.; the connecting discussion and worked transfer example are this note’s exposition.
Related notes: Deep information propagation · Tensor Programs IV & V · Tensor Programs VI.