Reading the order of a dead direction
Not every dead direction is equally dead, and that difference is the whole game. Real networks grow them everywhere, out of saturated units, collapsed features, and the sheer redundancy of overparameterisation, and the measure of the difference is the order $k$ from the previous explainer#3: the KL divergence to the truth climbs as
$$ K(w) = w^{2k} $$
when you step a distance $w$ from the singular point, so a $k = 2$ direction charges you by the fourth power of the step and a $k = 4$ direction by the eighth, a floor so flat the move is nearly free. How simple the model effectively is, and how well it should generalise, both hang on that integer. What this explainer adds is the part that makes it usable: how you could ever read $k$ off a real network.
Two views of the same number
The order sits in two research traditions that look unrelated until you line them up. Watanabe's singular learning theory reads it the way the equation above does, from the loss valley itself. Amari's information geometry never looks at the valley: it watches the Fisher information, the stiffness from the definition, and along a dead direction the rate theorem says the Fisher decays as
$$ F(w) \;=\; c\, w^{2(k-1)}, $$
a straight line of slope $2(k-1)$ on a log–log plot.1 The intuition for the exponent is that the Fisher tracks how sharply the KL curve bends, and bending a $w^{2k}$ valley twice brings the power down by two. One integer, two measurable exponents, read from two different surfaces. That coincidence is the bridge this whole program is built on, and the dead direction is the object that lives on both sides of it.
Slide $k$ and watch both exponents move together. Slide $w$ to walk toward the singular set and watch the Fisher information collapse. The case $k=1$ is the familiar, regular one: the Fisher information stays bounded away from zero (slope $0$), and ordinary statistics applies. Everything with $k \ge 2$ is a genuine dead direction.
Why the slope is the point
Reading $k$ the orthodox way means resolving the singularity with a change of coordinates (a Hironaka resolution), which is not something you can run on a trained transformer. The slope view sidesteps that entirely. If you can watch $F(w)$ shrink along a direction during training, the slope hands you $k$ directly, and from $k$ you get this direction's contribution to the local complexity that controls Bayesian generalization, its per-direction real log canonical threshold:
$$ \lambda = \frac{1}{2k} $$
| KL order $k$ | KL exponent $2k$ | Fisher slope $2(k-1)$ | RLCT $\lambda = 1/(2k)$ | regime |
|---|---|---|---|---|
| 1 | 2 | 0 | 0.500 | regular (non-degenerate) |
| 2 | 4 | 2 | 0.250 | singular |
| 3 | 6 | 4 | 0.167 | singular |
| 4 | 8 | 6 | 0.125 | singular |
A lower $\lambda$ means a simpler effective model occupying more parameter volume, which is the singular-learning version of Occam's razor. Measuring it from a slope is what makes the reading cheap enough to run at scale.
Many dead directions at once
So far we have read one dead direction at a time. A trained network never has just one. Every weight-space symmetry, every redundant or duplicated unit adds another, so dead directions come in bunches, and the number sharing the flattest order has a name: the multiplicity $m$.
One caveat about the name: Watanabe's own definition of $m$ is more abstract, the order of a pole of a certain zeta function, surfacing in his theory as a small correction term in the cost of a model.2 The concrete count used here is what that abstract definition works out to in the clean case, where the dead directions cross the singular point independently and share one flattest order. So here the count is the multiplicity, by a theorem; on messier geometry the two can part ways, which is a story for later.
You can watch them appear in the Fisher spectrum. At the start of training every direction looks ordinary. As the model trains and compresses, the dead directions give themselves away: their eigenvalues collapse toward zero, while the live ones settle but hold.
Slide training forward and a fixed number of eigenvalues fall away while the rest stay put. That number is the multiplicity. A real network can develop thousands of dead directions, and reading them needs nothing new: each one's order comes from how fast its eigenvalue collapses, and the multiplicity is simply how many collapse together. One caution travels with the simplicity: the clean count assumes the collapsing directions share one piece of the singular set at one order. When structures of different kinds and orders mix, as they do on a real network, the count needs the more careful assembly of adding up the orders#23.
How flat each dead direction is, and how many there are, together make up most of what separates one singular model's effective complexity from another's.
What the clean reading leaves out
Even with the order and its multiplicity in hand, the clean reading still rests on coordinates already aligned with the geometry, and real networks add two more complications that later explainers take up. The whole picture can be rotated by a symmetry of the loss (a gauge), so the direction you measure depends on the optimizer. And the rate has to be lifted through many layers at once, which is where the K-FAC bridge comes in.
Where the clean reading applies. The clean slope assumes the trajectory actually approaches the singular set. A network that is still absorbing its task has no singular structure formed yet, and a read that returns nothing there is the right answer about the network's phase, not a failure of the instrument (nothing to read yet#20). It also assumes the descent leaves the rate legible: an optimizer like Adam can leave it unreadable in closed form, and a finite, noisy spectrum can hide the asymptotic slope. What to do then is answered twice over later in the section, once by an optimizer that respects the symmetries#17 and once by a read that needs no descent at all#21, taken at a single frozen checkpoint in whatever coordinates the optimizer left.
- One warning about the intuition: at a singular point the Fisher is not the second derivative of the KL, and differentiating $K(w) = w^{2k}$ twice predicts the wrong constant, $2k(2k-1)$, where the true leading Fisher-to-KL ratio is $2k^2$ (they agree only in the regular case $k=1$). The exponent $2(k-1)$, which is all the slope reading uses, survives. The rate theorem, with the corrected ratio, is from Tejas Pradeep Shirodkar, Dead Directions: Geometric Singular Learning, arXiv:2606.05957 (2026). ↩︎
- Formally, $m$ is the order of $-\lambda$ as the largest pole of the zeta function $\zeta(z) = \int K(\theta)^z \pi(\theta)\, d\theta$, and it enters Watanabe's free-energy expansion as the $-(m-1)\log\log n$ term (Sumio Watanabe, Algebraic Geometry and Statistical Learning Theory, Cambridge University Press, 2009). That this pole order equals the flattest-order count for a normal-crossing singularity under a uniform prior is proved in Tejas Pradeep Shirodkar, Dead Directions: Geometric Singular Learning, arXiv:2606.05957 (2026). ↩︎