What is a dead direction

Take the smallest interesting network and try to change its weights without changing what it computes. The network is a single hidden ReLU unit with one weight on each side, $f(x) = c \cdot \mathrm{ReLU}(w \cdot x)$. There is a way, because ReLU is positively homogeneous: double the incoming weight $w$, halve the outgoing weight $c$, and the two moves cancel exactly. You have moved through weight space and the function has not noticed. Do it continuously and you trace out a whole path of weight settings that are all the same model. The direction along that path is your first dead direction: the loss is flat along it, because the model's sensitivity there is already gone.

Slide $\alpha$. The two weights trade off along the orbit $w\,c = \text{const}$, the hidden unit rescales its internal signal, and the output never moves. Every point on that orbit is the same model, and the move between them is a pure reparametrisation: $(w, c) = (1, 1)$ and $(2, \tfrac{1}{2})$ are two spellings of one function, the way ten millimetres and one centimetre are two spellings of one length. Only the description moved. Notice also what kind of thing this is: a property of a direction in weight space. It is a different animal from a dead neuron, a single unit stuck at zero output, and it can run along one neuron's axis or cut diagonally across several.

The rescale is the extreme case, perfectly flat forever, because nothing observable ever changes along it. The dead directions that carry information ask for less. Along them the function does still change, only with a sensitivity that dies away as the weights approach a special point, and how fast it dies is something you can measure. Information geometry supplies the measuring language: the Fisher metric. Written as a matrix, the Fisher $F(\theta)$ hands every direction $u$ a stiffness, the quadratic form $u^\top F u$: how sharply the predictions move if you take a small step along $u$. A live direction has stiffness; a dead one is a direction whose stiffness dies as you approach the point.

Definition (Dead direction)

A unit direction $u \in \mathbb{R}^d$ is a dead direction at a point $\theta_0$ when the Fisher information along it vanishes on approach, $u^\top F(\theta(t))\,u \to 0$ as $t \to 0$, for a path $\theta(t)$ leaving $\theta_0$ along $u$. Its KL order is the integer $k \ge 2$ for which the divergence to the truth behaves as $K(\theta(t)) = c\,t^{2k} + O(t^{2k+1})$ with $c > 0$, a zero of order $2k$.¹

In plain terms, $K$ is the Kullback–Leibler divergence: it measures how far the model has drifted from the truth after a step of size $t$ along the direction, and it starts to climb like $t^{2k}$. The order $k$ is just that power, and it makes sense for any direction. A regular direction has $k = 1$, so the loss climbs like $t^2$, the ordinary bowl, and its Fisher stiffness never dies, which is why the definition starts at $k = 2$: a dead direction climbs like $t^4$, $t^6$, or slower, and the larger $k$ is, the flatter the approach.

The exact symmetry that opened this explainer sits at the very top of that ladder. Its $K$ never climbs at all, so no finite $k$ fits it, and the theory files it as the limiting $k = \infty$ case, flat to every order. The definition's membership condition still covers it, a stiffness that is identically zero certainly counts as one that has died, but an order, the part that carries information, belongs only to the finite-$k$ deaths. That split, membership for both, an order for one, is worth holding onto; the last section below turns it into a picture.

The same symmetry, all over the network

Attention heads have their own version of this. A head's attention pattern depends only on the dot products between its query and key vectors, and a rotation preserves dot products, so you can spin the head's entire internal basis and it attends to exactly the same tokens.

And whenever two units in a layer compute the same feature, only their combined output weight matters, so the work can slide from one unit to the other for free.

The singular set

Each of these is one orbit: a path you can travel through weight space while the function holds still. Orbits run through every point of weight space, good fits and bad alike, since you can rescale or rotate any setting of the weights, and sliding along one never improves anything; it only rewrites the model you already have. The singular set is something more specific: the set of weights that exactly achieve the best fit. In an overparameterised network it is a surface rather than a point, the flat valley floor of the loss, and because a reparametrised copy of a perfect fit is still a perfect fit, the orbits of any point on the floor run along the floor. Here is the whole picture for the two-weight unit we started with:

Every faint curve is one orbit, two points on the same curve are two spellings of one function, and the plane is covered in them. Flatness along itself is ordinary, every orbit has it; what singles out the gold curve is its height, the one orbit lying at loss zero. A real network's floor is this same picture in billions of dimensions, a dense weave of orbits, folded and crossing through itself.

Along the floor, and across it

With the floor in view, the two kinds of flat direction this explainer has met stop being an abstract distinction and become two directions of travel, and that geography is the fact the rest of the site keeps coming back to.

Travel along the floor and you are inside the singular set the whole way, moving between reparametrisations of the same perfect fit. This is what the symmetry directions do, and the papers call them gauge directions, a name borrowed from physics for exactly this freedom of description. Nothing observable ever changes on the journey, so there is nothing to learn from it, and when the theory counts a model's genuine degrees of freedom these directions are quotiented away.

Step across the floor, off the singular set, and the loss has to rise, however slowly. The crossing directions are where the finite orders live: how slowly the loss rises is exactly the $t^4$, $t^6$ of the definition box, and that order carries all the information. This is the genuine death of a degree of freedom, the structure singular learning theory was built to measure.

So the floor sorts every flat direction by its geography: along it, a symmetry, divided out; across it, a dying degree of freedom, measured. Reading that order is where the section goes next, and much later, once the instruments are all on the table, the full zoo of ways a direction goes flat, and the read that tells every kind apart on a real network, gets an explainer of its own: every way a direction dies#22.

The definition and the rate theorem are from Tejas Pradeep Shirodkar, Dead Directions: Geometric Singular Learning, arXiv:2606.05957 (2026). ↩︎