The cost of reading a dead direction
Every read in this section pulls the same singular structure off a network, and no two charge the same price. The cheapest is a glance at a single parameter vector; the dearest in-house read is a full eigendecomposition; and beyond the ladder entirely sits the posterior sampler, thousands of times dearer again. The prices are not a defect to engineer away. Each read answers a different question, and the questions get harder in the same order the prices climb: is anything dead here, where is it, how many are dying together, and what is the one number the whole model earns. Pay for the question you are asking, and not a rung more.
The ladder, rung by rung
Start at the bottom, with the read that costs almost literally nothing. If the model uses LayerNorm, the kernel direction#10 $\gamma^{-1}$ is sitting in the scale parameter, and "reading" it means normalising a vector you already have on disk. No forward pass, no data, no gradients. What it buys is presence: this model carries that dead direction, guaranteed by the architecture and deepened by training. What it cannot tell you is anything about the structure training added, and it only exists on LayerNorm.
One forward pass up the ladder is $\sigma_{\min}$ of the activations#9: run a batch through, take the spectrum of what each layer produced, and the flattest direction of the representation falls out. It works on any architecture, it is cheap enough to run every few steps, and thanks to the residual stream its reading means the same thing at layer 3 and layer 30. This is the workhorse presence read, and, run over training, the when read: a $\sigma_{\min}$ still rising says there is no structure formed yet to look for (nothing to read yet#20).
The third rung crosses to the gradient side. One backward pass gives $u^\top G u$, the loss curvature along a particular direction $u$, which is how a dead direction gets localised: to a layer, to a sub-layer, to one mode of one factor. The catch is inside the letter $u$. A curvature probe is only as good as the candidate direction you feed it, and where the candidate comes from is the heart of the frozen-checkpoint read#21: at a trained checkpoint the two K-FAC factors carry enough information to construct the candidate outright, rather than hoping an eigensolve of a noisy bottom spectrum happens to surface it.
The top in-house rung is the whole spectrum at once: the active volume $\log\det^{+} G$, the sum of every live eigenvalue's log, whose decay slope counts the collapsing directions#8. This is the read that answers how many, the rank-deficit that a single eigenvalue is blind to, and it is the one rung whose price has a cliff in it: an eigendecomposition costs $O(h^3)$ in the layer width $h$, pocket change at toy scale and the dominant cost at transformer scale.
Off the top of the ladder
One read does not fit on the chart. The established estimator of the global learning coefficient, the SGLD-sampled local learning coefficient, does something none of the rungs do: it integrates over the posterior rather than probing a point, and it returns the one number $\lambda$ for the model as a whole. The price of that integration is $10^4$ to $10^6$ forward–backward passes per checkpoint, which works out to three or four orders of magnitude above the cheap rungs.1 The relationship is complementary, not competitive: the ladder reads are directional, layer-local, and cheap enough to travel with training, and when you want one integrated coefficient with the sampler's guarantees, the sampler remains the tool. Adding up the orders#23 makes the complement precise, including the cases where blind assembly of cheap reads genuinely cannot replace it.
The order is not fixed
Turn the width knob in the demo and the ladder reorders. The one-pass reads, $\sigma_{\min}$ and $u^\top G u$, scale roughly linearly in the width and stay cheap at every size. The full spectrum is actually cheap on a small network, cheaper than its reputation, but its $O(h^3)$ eigendecomposition compounds until, at transformer width, it has pulled hundreds of times above the one-pass reads.1 So "which read is expensive" is not a fact about the read; it is a fact about the read at a width.
Cheap enough to run live
Because $\sigma_{\min}$ and $u^\top G u$ cost only a pass or two at any width, they run at real-time cadence, every step of training if you like. The full spectrum runs periodically on a small network and offline at large width, and the sampler runs when a checkpoint deserves it. So the ladder is also a schedule: watch presence and location live, reach for the count at checkpoints, and reserve the posterior for the readings only it can give. One gate applies to all of them: an order is only there to be read once the structure has formed, so the live reads double as the trigger that says when the dearer ones are worth running (nothing to read yet#20).
The prices above assume you can trust what the reads return, and that is exactly where the next arc begins: the network's own symmetries can corrupt a reading in a way no amount of budget fixes, and the optimizer itself turns out to be part of the instrument#12.
Still open / underspecified. The per-checkpoint costs shown here are schematic forward-pass-equivalents; real wall-clock depends on batch size, hardware, and implementation. Mapping where a cheap read is tight enough to substitute for a dearer one, and where it only ranks models in the same order, is part of the measurement work.
- The per-checkpoint cost ordering, how it shifts with width, and the $10^3$–$10^4\times$ gap to the calibrated SGLD estimator (which itself needs $10^4$–$10^6$ passes per checkpoint) are from Tejas Pradeep Shirodkar and P. J. Narayanan, Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity, arXiv:2606.21158 (2026). ↩︎