An optimizer that respects the gauge
The drift was never hypothetical. Sliding along a gauge orbit tilts a dead direction's coordinates while the model holds perfectly still, and which point a run settles on is decided by its optimizer. The most widely used one settles badly. Adam scales each weight's step by its own recent gradient size, a separate $1/\sqrt{\hat v}$ for every coordinate, and a per-coordinate rule like that leaves a symmetry alone only when the symmetry just renames coordinates. The architectural gauges do more than shuffle. The ReLU rescale trades magnitude between a weight coming in and a weight going out; a rotation mixes an attention head's coordinates into sums and differences. A per-coordinate step cuts across that mixing, so part of every Adam update points along the orbit, the flat gauge direction, instead of across it toward lower loss.1
The leak has two costs, and both have already shown up. It moves the point the run settles on, choosing a particular representative of the function from among all the ones the orbit holds, often a more degenerate one. And it smears the reading: as the run slides along the orbit, the coordinate components of every dead direction it carries slide with it, so the order the geometry actually holds becomes harder to recover from the path.
Split the step
The fix follows the shape of the problem. Every update can be decomposed into two parts: the horizontal part, across orbits, which is the genuine descent through the quotient; and the vertical part, along the orbit, which is pure gauge freedom and changes nothing the network computes. Adam mixes the two because it conditions in raw coordinates.
DDC's construction keeps them apart, in three moves. It splits the incoming update against a metric the gauge leaves invariant, so "along the orbit" and "across it" mean the same thing everywhere. It conditions each part on its own terms: the horizontal part keeps Adam's full per-coordinate adaptivity, read in a frame that turns with the symmetry, while the vertical part is collapsed to one controlled scalar per gauge dimension, since a flat direction needs bookkeeping rather than adaptivity. Then it re-projects the conditioned step back across the orbits, so what actually lands is a step on the quotient.
The trajectory is then a preconditioned gradient flow on $\bar\Theta = \Theta/G$, the space of distinct functions, with no component along the gauge. The run no longer slides, so the representative it settles on and the dead directions it reads stop depending on where it happened to start on the orbit.
Scaling is simple, rotation needs a frame
How hard the split is depends on the gauge. The scaling and shift gauges, the cross-entropy logit shift, the ReLU and SwiGLU rescaling, the LayerNorm and RMSNorm scale, are each a single number per orbit dimension, and they commute, so their order of application does not matter. One scalar per channel conditions them exactly, and the construction is provably gauge-equivariant for them down to floating-point precision.
The per-head attention rotation is harder. Two rotations of a head compose in an order-dependent way and mix the head's coordinates, so a per-coordinate estimate $\hat v$ (Adam's running record of each coordinate's recent gradient size) read in the fixed axes is itself gauge-dependent. The cure is to condition in the head's own body frame, the eigenbasis of $W^\top W$, keeping full adaptivity on the live directions and a single conditioned scalar on the dead ones. Because the frame turns with the rotation, the conditioning means the same thing at every point on the orbit, which is what equivariance asks for; exactly in principle, and to leading order in the running implementation, where the frame is re-estimated as the weights move. The coarser per-head variants, one scalar or one small matrix per head, close the symmetry exactly at the price of less adaptivity.
The split does not care which optimizer it wraps
The decomposition sits on top of any preconditioner, so the same construction lifts Adam and Muon alike. What changes between them is which gauges each base can carry. A purely orthogonalising optimizer already commutes with rotation, since rotating a weight rotates its orthogonalised update by the same amount, so it carries the head-rotation gauge exactly. It also erases per-channel scale outright, so the scaling gauges have nothing left to condition on that base and ride the Adam base instead. The optimizer is the vehicle here. Respecting the gauge is the part that matters, and it is worth the trouble for two reasons: the minimum the run reaches, and what that minimum leaves you able to measure. The next explainer#18 pays out both.
Still open / underspecified. The construction is proved per gauge family, and its edges are visible. Composing the non-abelian rotation gauge with the abelian ones on a single base is handled by giving each base the gauges it carries natively, not by one construction that carries all of them. The running body frame is equivariant to leading order, with a small finite-precision drift that accumulates over steps. And which gauges engage on a given architecture, whether a nominally present symmetry actually has weight-space room to drift in, is an empirical question the experiments have to check case by case.
- The horizontal–vertical construction, its equivariance proof, and the experiments behind the next reading are from Tejas Pradeep Shirodkar, Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks, arXiv:2606.29176 (2026). ↩︎