Parameter-readable kernel directions
Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale · arXiv:2606.194911 · retitled for the next revision to Parameter-Readable Kernel Directions in LayerNorm Transformers, which is the framing this page uses
The cheapest read in the program, taken to the largest models. The paper proves that LayerNorm's mean-centring pins one exactly flat direction of the representation into every model that uses it, the reciprocal scale vector $\gamma^{-1}$, and that RMSNorm, which skips the centring, provably creates none. The identity is activation-side; in weight space the same direction has the ordinary KL order $1$, with its Fisher strongly suppressed and the order untouched, and the paper carries that as scope. That yields predictions sharp enough to check to the decimal, and the check runs twice over: at random initialisation, before any training, the predicted direction already matches the measured flattest direction at $|\cos| > 0.9999$ on all nine LayerNorm architectures, and on the pretrained releases themselves the match only tightens, while on five RMSNorm models it is correctly absent, a normalisation classifier that never runs the network. Alongside the kernel sit the residual-stream depth-invariance result, thirteen of fourteen models holding the $\sigma_{\min}$ floor block by block, and the paper's best story: the one exception was not a failed prediction but a genuine dead direction in that model, which the same reads pinpointed to a sub-layer.
It is also the paper of measurement discipline: reference points, precision floors, and calibration distributions, the craft that makes a four-decimal claim at 31 billion parameters meaningful.
Where it is explained here
- The kernel and the classifier: reading a kernel direction off the weights#10
- The depth-invariant floor: a floor that survives depth#9
The signature demo
The fourteen-model survey itself:
Of the five, this is the one that carries the geometry to production scale: the same object the theory defines, located by hand inside a 31-billion-parameter model.
- Tejas Pradeep Shirodkar and P. J. Narayanan, Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale, arXiv:2606.19491 (2026). ↩︎