Reading a dead direction off the weights
The cheapest read of a dead direction needs no forward pass. The earlier reads found dead directions by running the network: the K-FAC bridge needs a layer's curvature, and $\sigma_{\min}$ needs a forward pass through the activations. For one common component, a LayerNorm, you can do better still and read a dead direction straight off a single parameter vector, without running the model on anything.
What centring does to a direction
A LayerNorm does two things to its input: it centres it, subtracting the mean across the features, and then scales each feature by a learned gain $\gamma$. The centring is what matters. Subtracting the mean forces every normalised vector to sum to zero, so it lands in the flat hyperplane of vectors whose components cancel. The all-ones direction, the one every centred vector is orthogonal to, carries no variance whatsoever: the output simply never moves along it.
The $\gamma$ scale then tilts that fixed dead direction. After scaling, the direction the output cannot move along is the reciprocal $\gamma^{-1}$, one over each gain, normalised. The check is a single line: the output is $\gamma \odot \hat{x}$, and its dot product with $\gamma^{-1}$ is $\sum_i \hat{x}_i$, which the centring pinned to zero, for every input. So the representation is exactly orthogonal to $\gamma^{-1}$ whatever you feed it. That is a dead direction, sitting in the scale parameter in plain sight. Directions a map crushes to nothing form what linear algebra calls its kernel, which is why this one is called the LN-kernel direction.
Move the $\gamma$ sliders and the dead direction $\gamma^{-1}$ moves with them, always dead: the variance of the representation along it stays at zero to machine precision. You never ran the network. You read the direction off the weights, and the centring guarantees the rest.
RMSNorm has no such direction
RMSNorm is a LayerNorm with the centring removed: it divides by the root-mean-square and never subtracts the mean. With no centring there is no hyperplane that every vector lands in, so no direction is universally starved of variance, and $\gamma^{-1}$ becomes an ordinary direction with ordinary spread. Switch the centring off in the demo and the kernel vanishes. This buys a free classifier: whether the $\gamma^{-1}$ dead direction exists tells you if a model normalises with LayerNorm or RMSNorm, from its parameters alone.
At random initialisation, before a single step of training, the predicted direction already matches the network's measured flattest direction to four decimal places across LayerNorm architectures, and it is correctly absent in RMSNorm models. Training then deepens what the architecture began: the variance along $\gamma^{-1}$ collapses by orders of magnitude more, turning the kernel the model was born with into one of its deepest dead directions.1 (The depth-invariant floor of the previous explainer#9 is a separate mechanism, the identity skip, and holds under either normalisation.)
Here is the survey itself, fourteen pretrained transformers from 160 million to 31 billion parameters, language and vision:
The kernel also turns out to be more than a curiosity of one layer type. On a fine-tuned vision transformer examined direction by direction, every dead direction the model carries is of this symmetry kind, the LN-kernel and its attention-rotation cousins, with no genuinely dying feature anywhere in the network. Which kind of dead structure a model carries, and how to tell, is taken up in every way a direction dies#22.
Still open / underspecified. The kernel is exact at the final normalisation, where centring is the last thing applied. Deeper inside the stack, biases, the next block's input projection, and finite precision can move the measured flattest direction a little off $\gamma^{-1}$; pinning down how far that interior drift goes, sublayer by sublayer, is part of the measurement work.
- The algebraic kernel $\gamma^{-1}/\|\gamma^{-1}\|$ and its validation across pretrained LayerNorm and RMSNorm transformers are from Tejas Pradeep Shirodkar and P. J. Narayanan, Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale, arXiv:2606.19491 (2026). ↩︎