Why language models are trained with AdamW, and what is changing

Four labs that share no data, no architecture, no hardware and no opinion on much of anything published the same paragraph between 2020 and 2024. GPT-3 was trained with Adam, $\beta_1 = 0.9$, $\beta_2 = 0.95$ and weight decay $0.1$. OPT, with AdamW, $0.9$, $0.95$ and $0.1$. Llama 2, with AdamW, $0.9$, $0.95$ and $0.1$. DeepSeek-V3, with AdamW, $0.9$, $0.95$ and $0.1$. The update rule fits in four lines, it was published in 2017 as a fix to another one from 2014, and the three numbers that go with it haven't moved in five years or across continents.

And then, within fourteen months, three cracks appeared. Kimi K2, in July 2025: a trillion parameters trained without AdamW. GLM-5, in February 2026. DeepSeek-V4, in April 2026, with 1.6 trillion parameters: the same lab that sixteen months earlier had written everybody's paragraph. All three use Muon, an optimiser that in December 2024 was a blog post.

GPT-3 calls it Adam and cites the AdamW paper for the decay: it's the same algorithm. And the three at the bottom haven't dropped AdamW entirely. DeepSeek-V4 keeps it for the embeddings, the output layer, the biases and the normalisations, and uses Muon for everything that's a matrix. That split isn't an implementation detail: it's the idea behind Muon, and this article ends on it.

The story starts much earlier. Mathematical optimisation has spent two centuries accumulating methods, and a good share of them beat gradient descent on almost any criterion you care to measure: steps to arrive, sensitivity to the settings, guarantees. Newton's method, which is more than three centuries old, solves in a single step a problem that gradient descent needs thousands of steps for. Nobody trains with Newton. This article asks why, and the answer has three parts: why, at this scale, the only information you can pay for is the gradient; how each improvement to the descent buys back, on credit, something Newton had for free; and why the latest of those improvements is the one that has moved the throne.

The order will be logical rather than chronological: each method fixes one specific failure of the previous one, and that sequence isn't the order of the dates. To keep the dates in view:

What there is to minimise, and what a step can cost

Let's call $\theta$ all the parameters of the model, stacked into a single vector with $P$ coordinates, and $\mathcal{L}(\theta)$ the loss: a number that measures how wrong the model is with those parameters, and which training aims to make small. For Llama 3 405B, $P$ is $4.05 \cdot 10^{11}$, and the loss is a mean over $15.6$ trillion tokens:

where $\ell_i$ is how wrong the model is on token $i$ and $N$ is those $15.6$ trillion. Three things about that definition decide everything that follows.

The first is that $\mathcal{L}$ can't be evaluated. Computing it exactly means running the whole model over the $N$ tokens once, and that is, to a reasonable approximation, what the whole training run costs: a language model sees each token of the corpus once, or not much more. Evaluating the exact loss a single time costs as much as training. Any method that asks for the exact value of $\mathcal{L}$ at every step is ruled out before it starts.

The second is that the minimum isn't needed. $\mathcal{L}$ isn't convex: it has many valleys, and no method you can run guarantees finding the deepest one. But that isn't what's being asked, either. What gets measured is the loss of the model you ship, and a valley that isn't the deepest will do if it's deep enough. That concession is what makes everything else possible.

The third is the price of each kind of information, and it's worth putting numbers on it. Evaluating $\mathcal{L}$ on a batch of data is one forward pass. Computing its gradient, the vector of $P$ partial derivatives that says which way is downhill from here, costs roughly two more forwards, using the chain rule run backwards: three forwards in total, plus a vector of $P$ numbers that takes as much room as the model. Computing the matrix of second derivatives, the Hessian $H$, which says how much the loss curves in each direction, produces $P^{2}$ numbers. For Llama 3, that's $1.6 \cdot 10^{23}$. The $16\,384$ H100 GPUs used to train it add up to $1.3 \cdot 10^{15}$ bytes of memory, and storing that Hessian in the lowest precision anyone trains in would take $250$ million times that. You don't even need the whole matrix for things to go wrong: the product of $H$ with a single vector, which gives the curvature in one direction only, costs about as much as an entire gradient.

That's the asymmetry everything comes from. With one gradient you know which way is down in all $P$ directions at once, and it costs three forwards. Knowing how much the loss curves in one direction costs the same, and knowing it in all of them fits nowhere.

The catalogue, ordered by what each method asks for

Mathematical optimisation methods can be sorted into four families by the kind of information they demand about the loss in exchange for a step: values of the function only, a promise about its shape, its second derivatives, or its first derivatives only. This section follows that order, leaving for last the family that training actually uses.

Derivative-free. Grid search, random search, the Nelder-Mead simplex of 1965, genetic algorithms, evolution strategies, Bayesian optimisation. They all ask for the same thing, which is the minimum: evaluate $\mathcal{L}$ at points of their choosing. They differentiate nothing, and in exchange they have to work out where the bottom is by comparing values, and in $P$ dimensions there are $P$ independent directions to compare. Approximating the gradient by finite differences costs $P + 1$ evaluations, against the three it costs to compute it exactly. Evolution strategies estimate it more cheaply, perturbing the parameters at random and keeping the perturbations that lower the loss, and in 2017 they taught a simulated humanoid to walk in ten minutes by spreading the work over more than a thousand processors. But the noise of that estimate grows with $P$, and $P$ has grown a hundred thousand times since. What you save by not differentiating, you pay for $P$ times over in evaluations.

Structured. Dantzig's simplex of 1947 and the interior-point methods solve linear programs with hundreds of thousands of variables; quadratic programming and the whole of convex optimisation, with its subgradients, its proximal methods and its guarantee that the minimum found is the best there is, solve enormous problems in seconds. What they ask in return isn't information about $\mathcal{L}$ but a promise about its shape: that it be linear, quadratic or convex. A network's loss is none of the three, and there's no way to make it so.

Second order. Here is the method that solves the problem, and it's worth seeing why. Near $\theta$, the loss looks like its second-order approximation, a quadratic with the gradient as the linear term and the Hessian as the curvature:

If that approximation were exact, the best step would be the one that minimises it. Differentiating with respect to $\Delta$ and setting the result to zero gives $\nabla_{\theta}\mathcal{L} + H\Delta = 0$, that is,

which is Newton's method. It has no learning rate: the size of the step is decided by the curvature, and decided separately in each direction. Where the loss curves a lot, the step is short, and where it curves little, the step is long, without anyone having to say so. Newton has for free what everything else is going to have to pay for: the curvature of every direction.

What it costs has already been said: $P^{2}$ numbers to store $H$, and on the order of $P^{3}$ operations to solve the system. The quasi-Newton methods were born to avoid paying that. BFGS, named after its authors Broyden, Fletcher, Goldfarb and Shanno, is one of the best known. Instead of computing the Hessian, it builds an approximation to $H^{-1}$ from how the gradient changes from one step to the next, which is information you already have. It still takes $P^{2}$. L-BFGS (limited-memory BFGS), proposed later, avoids storing the whole approximation to the inverse Hessian: it keeps only the last $m$ pairs of changes, with $m$ between five and twenty, and from those it rebuilds the product $H^{-1}\nabla_{\theta}\mathcal{L}$ in $4mP$ operations and $2mP$ of memory. That cost is affordable, and for twenty years L-BFGS was machine learning's default: logistic regression, conditional random fields, support vector machines with a linear kernel. In 2011 it was tried on deep networks, and on some problems it held its own against gradient descent.

It stumbled on two things it needs that a batch of data can't give it. The first: the pair of changes it uses to update its approximation, how far $\theta$ moved and how much the gradient changed, only makes sense if both gradients were computed on the same data; with different batches, the difference between the gradients is mostly the difference between the batches, and that difference goes into the estimate of $H^{-1}$ as if it were curvature. The second: to decide how far to go in the direction it proposes, L-BFGS runs a line search, which tries several step sizes and keeps one that lowers the loss enough, and to know whether the loss has gone down, you have to be able to evaluate it. Both need exact values, and the first section said what one of those costs. The Hessian-free methods of 2010 tried the other route: don't store $H$, use only $Hv$ products, each at the price of a gradient, to solve Newton's system by conjugate gradient. That came to tens or hundreds of products per step. In the same time, gradient descent takes tens or hundreds of steps.

First order. What's left. One gradient per step, three forwards, $P$ numbers of memory, and no promise about the shape of $\mathcal{L}$.

The last column is the one that decides, and it deserves its own paragraph, because the reason gradient descent doesn't merely survive but suffices lies there, not in the cost. The loss is a mean of $N$ terms, so its gradient is the mean of $N$ gradients, and the mean of a handful of them chosen at random is an unbiased estimator of the mean of all of them: it points, on average, at exactly the same place, and it costs only what the handful costs. The Hessian can be estimated that way too. What can't be estimated that way is its inverse, which is what Newton needs: the inverse of a noisy estimate isn't an estimate of the inverse, and the noise in the gently curved directions, which are the ones with small entries in $H$, comes out amplified by the inversion. The gradient is the one piece of information that is both cheap and estimable without bias from a slice of the data.

Everything from here on can be read this way: gradient descent is the only thing you can pay for, and each of its improvements is a way of approximating $H^{-1}$ that survives the noise. The diagonal, in RMSProp and Adam. A product of small matrices, in Shampoo. The spectrum of each weight matrix, in Muon. Each buys back a piece of what Newton had for free, at a price you can afford.

Gradient descent, and the one number you choose

In 1847paperCauchy and the Gradient MethodClaude Lemaréchal, 2012 · Documenta Mathematica, Extra Volume ISMPFour pages on Cauchy's 1847 note proposing the method, with the key passage translated. It shows the method was invented to solve systems of equations, not to minimise anything, and that the learning rate already worried its author., Cauchy presented to the Paris Academy of Sciences a three-page note on solving, without isolating any unknowns, the systems of equations that give the orbit of a heavenly body. The idea fits in a sentence: wherever you are, look at which way the function climbs fastest and move a little the opposite way. What takes a whole article is working out what "a little" means.

The gradient $\nabla_{\theta}\mathcal{L} \in \mathbb{R}^{P}$ collects one partial derivative per parameter: each coordinate says how fast the loss changes when that parameter moves and the others stay put. That it points in the direction of fastest ascent isn't a definition but a theorem, and it follows from an inequality. Take any direction $\mathbf{u}$ of length $1$; how much the loss changes when you move a little along it is the directional derivative, which is a dot product, $\left(\nabla_{\theta}\mathcal{L}\right)^{\top}\mathbf{u}$, and by Cauchy-Schwarz that product is bounded by $\lVert \nabla_{\theta}\mathcal{L} \rVert$, with equality only when $\mathbf{u}$ is parallel to the gradient. No direction climbs more per unit of step than the gradient's, so none descends more than the opposite one. From there, the whole rule follows:

where $t$ counts steps and $\eta > 0$ is the learning rate, the one number in the formula that you choose. You can see why every step lowers the loss by using the tangent plane:

and with $\Delta = -\eta\,\nabla_{\theta}\mathcal{L}$, the loss drops by $\eta\,\lVert\nabla_{\theta}\mathcal{L}\rVert^{2}$, which is never negative. The whole debt is in the $\approx$: it holds near $\theta_t$, and how near is decided by $\eta$.

The arithmetic comes out whole on a parabola

Take the smallest loss there is, a single parameter and $\mathcal{L}(w) = w^{2}$, with derivative $2w$. The rule becomes $w_{t+1} = w_t - 2\eta\,w_t = (1 - 2\eta)\,w_t$: multiply by the same constant at every step. Iterating from $w_0$ gives

and a power tends to zero if and only if its base has absolute value less than $1$. The condition is $\lvert 1 - 2\eta \rvert < 1$, that is, $0 < \eta < 1$, and the factor itself tells you what happens in each range: between $0$ and $0.5$, the weight shrinks without changing sign; at $0.5$, it lands on the minimum in a single step; between $0.5$ and $1$, it alternates sign and shrinks; at $1$, it alternates forever; above that, it alternates and grows.

That limit of $1$ belongs to this particular parabola. With $\mathcal{L}(w) = a\,w^{2}$, the derivative is $2aw$, the factor becomes $1 - 2\eta a$ and the condition becomes $\eta < 1/a$: the more the loss curves, the shorter the step has to be. So far nothing hurts, because $a$ is a single number and $\eta$ can be chosen by looking at it.

One rate for two curvatures

It hurts as soon as two directions curve differently, and a network has $P$ of them. The function $\mathcal{L} = x^{2} + 20y^{2}$ is the least you need to see it: an elongated ravine, nearly flat along $x$ and very steep along $y$. The parabola's arithmetic, coordinate by coordinate, asks for $\eta < 1$ for $x$ and $\eta < 1/20 = 0.05$ for $y$, and the second constraint rules. The per-step factors are $1 - 2\eta$ in $x$ and $1 - 40\eta$ in $y$.

The two trajectories are the same dilemma seen from opposite sides. A small rate respects the steep direction and wastes the flat one; a large rate exploits the flat direction and bounces around in the steep one. No $\eta$ serves both well, because $\eta$ is one number and the curvatures are two.

The ratio between the largest curvature and the smallest is called the condition number, $\kappa$, and here it's $20$. Let's call $a_{\max}$ the curvature in the steepest direction and $a_{\min}$ the curvature in the flattest. The condition number governs the number of steps: the best possible $\eta$ is on the order of $1/a_{\max}$, and with it the flat direction contracts by a factor of $1 - a_{\min}/a_{\max}$ per step, so reaching a given precision takes a number of steps proportional to $\kappa$. In a network, you can't afford to compute that $\kappa$, but one measurement repeats in every model anyone has looked at: the most and least curved directions differ by several orders of magnitude. In a ravine with $\kappa = 20$, we're talking about a few tens of steps. A ravine a hundred times narrower, in the Adam section, will cost $3\,619$.

The problem has a name, ill-conditioning, and everything after this section is a way of treating it. Newton treats it by dividing by $H$, which is unaffordable. The rest are ways of approximating that division with what is affordable.

Stochastic: the gradient nobody can compute in full

Before treating the ravine, there's the other problem from the first section to deal with, and it comes first: the exact gradient costs as much as the whole training run, so nobody computes it. What gets computed is the gradient over a batch of $B$ randomly chosen examples,

and that is what gets subtracted. It's an estimator: its mean over all possible batches is exactly $\nabla_{\theta}\mathcal{L}$, because the gradient of a mean is the mean of the gradients, and its variance shrinks like $1/B$. It costs what $B$ costs, not what $N$ costs. For Kimi K2, each batch is $67$ million tokens out of a corpus of $15.5$ trillion: each step looks at $0.0004$% of the data and points, on average, in the same direction it would if it looked at all of it. For the compute of a single exact step, you get two hundred thousand noisy ones, and the noise averages out along the way.

That's stochastic gradient descent, SGD, and it isn't an approximation of the method but a theorem from 1951paperA Stochastic Approximation MethodHerbert Robbins and Sutton Monro, 1951 · Annals of Mathematical StatisticsThe theorem that lets you train with a gradient you never compute in full. Its two conditions on the rate (the sum diverges, the sum of squares doesn't) are the origin of every learning-rate schedule. about how to find the zero of a function you can only observe with noise. Robbins and Monro proved that the iteration converges if the rate decreases in a specific way: the sum of all the $\eta_t$ has to diverge, so the method can travel any distance, and the sum of their squares has to converge, so the noise eventually dies away.

The noise Robbins and Monro talk about is, in our case, the difference between the gradient we compute and the one we'd like to compute: the batch's minus the exact one,

It has mean zero, so on average we point in the right direction, but it isn't free. It costs three things, and all three matter later. The first is that an individual step can raise the loss; the descent guarantee from the previous section becomes a guarantee on average. The second is that near the bottom the trajectory doesn't stop: it trembles with an amplitude proportional to $\eta$ times the size of the noise, and the only way to settle it is to lower $\eta$. That's why no model trains at a constant rate to the end. The third is the one that closed the door in the previous section: with a noisy gradient, the line search can't tell whether the loss went down, and the secant condition measures noise. The second-order methods didn't lose on cost alone; they lost because the noise contaminates the curvature estimate, and the convergence guarantees that survive that noise are the first-order ones.

There's one thing the noise gives you, and it's worth stating carefully because it has often been exaggerated. A saddle point, where the gradient is zero but the loss goes down in some direction, is an unstable equilibrium: exact descent stays parked on top of it, while the noisy kind needs only a nudge in any downhill direction to leave. In high dimension, almost all the points where the gradient vanishes are saddles rather than minima, so this isn't a detail. But it isn't what makes SGD work; it's what keeps it from getting stuck.

With that, the method that trains every model in the table is complete: a noisy gradient, subtracted at a decreasing rate. What's missing is making it bearable in a ravine, and that's where the variants begin.

Momentum: remembering the previous step

The ravine's zigzag has a property you can exploit: in the steep direction, the gradient changes sign at every step, and in the flat one it keeps its sign. A method that averages recent gradients cancels the first and accumulates the second, without knowing which direction is which. Polyak proposed it in 1964 with a physical image, a heavy ball rolling over the surface and keeping part of its velocity, hence the name:

with $\beta$ between $0$ and $1$, and nearly always $0.9$. To keep things short, we write $\mathbf{g}_{t} = \nabla_{\theta}\mathcal{L}(\theta_t)$. Unrolling the first line, $\mathbf{v}_t$ is a sum of all the previous gradients with geometrically decaying weights:

With that, the arithmetic of the two directions comes out cleanly. If the gradient is always the same, $\mathbf{g}$, the sum tends to $\mathbf{g}/(1 - \beta)$: with $\beta = 0.9$, ten times the gradient, and the effective step is ten times $\eta$. If it alternates between $\mathbf{g}$ and $-\mathbf{g}$, the sum tends to $\mathbf{g}/(1 + \beta)$: a little more than half. The same rule amplifies tenfold the direction the gradient insists on and roughly halves the one where it contradicts itself. Averaging over time tells apart what the learning rate couldn't.

What doesn't change is that there's still a single $\eta$. Momentum doesn't remove the steep direction's limit; it moves it: with well-chosen parameters, the number of steps to a given precision goes from proportional to $\kappa$ to proportional to $\sqrt{\kappa}$, and the well-chosen parameters depend on $\kappa$, which nobody knows.

In 1983, Nesterov proposed a variant that evaluates the gradient not where the ball is but where it's going to be after the push, a step of anticipation that improves the theoretical guarantee, and in 2013 it was shown to train deep networks better. The difference from Polyak's version is small in practice; it matters in the last section, because Muon uses Nesterov's.

One rate per parameter: AdaGrad and RMSProp

The other attack on the ravine doesn't average over time; it rescales in space: instead of one $\eta$, use one per parameter, and let the method itself choose them. The rule is to divide each coordinate of the gradient by a measure of how large that coordinate's gradients have been. AdaGrad, from 2011, accumulates their squares:

where the square, the root and the division are all coordinate by coordinate, and $\varepsilon$ is a tiny number that avoids dividing by zero. It was born for problems with rare inputs. Think of a language model: the embedding row of a word only receives gradient at the steps where the word appears, and most words appear rarely. With a shared $\eta$, the frequent words set the limit and the rare ones barely move. With AdaGrad, each parameter has its own $\mathbf{s}$: those that receive gradient at every step accumulate it fast and slow down, and those that receive it once every thousand steps keep taking large steps. This will come back later, because it's a good part of the reason a language model isn't trained with plain SGD but with methods that adapt the rate parameter by parameter.

The flaw is in plain sight: $\mathbf{s}$ only grows, so the effective rate only falls, and on a non-convex problem, where you have to keep moving long after the start, it ends up stopping before it arrives. RMSProp fixes that by swapping the sum for an exponential moving average, which forgets:

That average's memory is about $1/(1 - \beta_2)$ steps: ten with $\beta_2 = 0.9$, a thousand with $0.999$. RMSProp was never published. It's a slide from a 2012 online course by Hinton, and for years it was cited that way, as "Lecture 6.5", because there was nothing else to cite.

What it does on the ravine, $\mathcal{L} = x^{2} + 20y^{2}$, can be seen without drawing it. In the steep direction the gradient is $40y$; in the flat one, $2x$. Dividing each by the root of its own mean of squares, which is more or less its magnitude, leaves each coordinate with a step of size $\eta$ and the gradient's sign, whatever its scale: the step has stopped depending on the curvature. That's exactly a diagonal approximation of $H^{-1}$, with the gradient magnitudes standing in for the curvature, and it costs one more vector of memory. Second-order information comes back in through the window, as a diagonal, at a price you can afford.

It has a start-up defect worth measuring, because it's the one Adam is going to correct. At the first step, $\mathbf{v}_1 = (1 - \beta_2)\,\mathbf{g}_0^{2}$, so the division leaves $\eta / \sqrt{1 - \beta_2}$ per coordinate: with $\beta_2 = 0.9$, that's $3.2\,\eta$, and with $0.999$ it's $31.6\,\eta$. On the ravine, with $\eta = 0.1$ and $\beta_2 = 0.999$, the first step takes the trajectory from $(-2.6,\ 1.2)$ to $(0.56,\ -1.96)$: it crosses the bottom and comes out the opposite wall, because the moving average hasn't seen anything yet and underestimates the size of the gradient thirty times over.

Adam: both ideas together

AdampaperAdam: A Method for Stochastic OptimizationDiederik Kingma and Jimmy Ba, 2014 · arXiv:1412.6980The original paper. The part worth reading is section 2.1, where the authors derive the bias correction and explain the step as a signal-to-noise ratio, which is what makes the learning rate independent of the gradient's scale., from December 2014, combines the two: a moving average of the gradient, which is momentum written as an average, and a moving average of its square, which is RMSProp.

And it corrects the start. Unrolling $\mathbf{m}_t$ as before, it's $(1 - \beta_1)$ times the sum of the gradients with weights $\beta_1^{t-k}$; if all the gradients were equal to $\mathbf{g}$, the geometric sum gives $\mathbf{m}_t = \left(1 - \beta_1^{t}\right)\mathbf{g}$. At the first step, the average is $(1 - \beta_1)\,\mathbf{g}$, a tenth of the gradient; at step $t$, it's a fraction $1 - \beta_1^{t}$ that climbs towards $1$. Dividing by that fraction restores the correct scale from the first step, and the same goes for $\mathbf{v}_t$ with $\beta_2$:

With that correction, the first step is $\eta$ per coordinate, with the gradient's sign, rather than $31.6\,\eta$. Those are the four lines mentioned at the top, plus two more vectors of memory, $\mathbf{m}$ and $\mathbf{v}$, each the size of the model.

The explanation of why it works on almost everything lies in the quotient. $\hat{\mathbf{m}}_t$ estimates the mean of the gradient and $\sqrt{\hat{\mathbf{v}}_t}$ estimates the root of the mean of its square; for those exact means, the first never exceeds the second in absolute value (variance can't be negative), so the quotient lies, coordinate by coordinate, between $-1$ and $1$: near $\pm 1$ where the gradient insists on a sign, near $0$ where its mean is noise. Adam's moving averages only approach that ideal, and the resulting step is almost always on the order of $\eta$, whatever the scale of the gradient; the exception is a gradient that sits at zero for many steps and then fires, where $\hat{\mathbf{v}}_t$ lags behind and the quotient can exceed $1$ (Kingma and Ba bound the step at $\eta \cdot \frac{1-\beta_1}{\sqrt{1-\beta_2}}$, about $3.16\,\eta$ at the default values). That's why $\eta$ can be chosen in units of the parameter, why the same $10^{-4}$ works for models that have nothing in common, and why a rate of $3 \cdot 10^{-4}$ means, almost always, that no weight changes by more than three ten-thousandths per step.

Look at the steps to the bottom before drawing conclusions: Adam takes $36$ and plain descent takes $35$. On a ravine with $\kappa = 20$ and a well-chosen rate, Adam isn't faster. Its advantage lies elsewhere, and to see it you have to change the ravine without changing the rate.

That's what Adam buys: not speed on a known problem, but independence from the curvature on an unknown one. A network has millions of ravines of different widths inside it, and nobody is going to measure any of them.

The W: a regularisation mistake that lasted three years

Almost every model is trained with weight decay: at every step, besides subtracting the gradient, each weight gets shrunk a little towards zero, $\theta \leftarrow \theta - \eta\lambda\,\theta$, so that none of them grows without bound. Under gradient descent, that's identical to adding a term $\tfrac{\lambda}{2}\lVert\theta\rVert^{2}$ to the loss, the L2 regularisation, because the gradient of that term is $\lambda\theta$ and subtracting it with $\eta$ gives the same formula. Two names, one algorithm, and for decades nobody had to tell them apart.

Under Adam, they stop being the same thing, and the arithmetic shows it. If the decay enters as a term of the loss, it enters $\mathbf{g}$, and from there $\mathbf{m}$ and $\mathbf{v}$: it goes through the division. For a parameter with historically large gradients, $\sqrt{\hat{\mathbf{v}}}$ is large and its $\lambda\theta$ comes out divided by a large number: it barely decays. For one with small gradients, it decays in full. The strength of the regularisation ends up depending on each weight's gradient history, and in the opposite direction to the sensible one: the weights that move the most are the ones held back the least.

Loshchilov and HutterpaperDecoupled Weight Decay RegularizationIlya Loshchilov and Frank Hutter, 2017 · arXiv:1711.05101The W in AdamW. Two pages of algebra show that L2 regularisation and weight decay are the same thing under SGD but different things under Adam, and that the one in common use was the wrong one. pointed it out in November 2017 and proposed taking the decay out of the division:

That's AdamW, and the W stands for weight decay, decoupled. The difference from the previous formula is where $\lambda\theta_t$ sits: outside the quotient, instead of inside $\mathbf{g}$. At the time, a result from that same year claimed that Adam generalised worse than SGD with momentum in vision, and a good part of that gap came down to this: Adam was regularising itself badly. With the decoupled decay, the gap closed on the tasks the authors tried. The $0.1$ in the table at the top is the $\lambda$ of this formula, and it gets multiplied by $\eta$: with $\eta = 3 \cdot 10^{-4}$, each weight shrinks by three hundred-thousandths per step.

Why AdamW, and why with those numbers

So far, the generic answer: Adam combines the two ideas and AdamW fixes its regularisation. The concrete question is why a transformer isn't trained with SGD with momentum, which worked in vision for years and keeps a single state vector instead of two, and the answer was slow to arrive.

It lies in the rare words, which make up almost the whole vocabulary. A language model's loss is a mean over tokens, and tokens follow Zipf's law: a few appear in every batch, and the vast majority appear once every many thousands. The parameters that touch only a rare token, its row in the embedding matrix and its row in the output layer, receive minuscule gradients on average, and under SGD a minuscule gradient means a minuscule step. Those parameters don't learn, and the loss on the rare tokens, which are most of the vocabulary, doesn't go down. Kunstner and colleaguespaperHeavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language ModelsKunstner, Yadav, Milligan, Schmidt and Bietti, 2024 · arXiv:2402.19449The cleanest explanation of why SGD can't train a language model and Adam can: rare words, which make up almost the whole vocabulary, get minuscule gradients, and Adam is the one that scales them separately. measured this in 2024, class by class, and it's exactly AdaGrad's problem: one rate per parameter is what lets a rare word's row advance as much per step as a frequent one's. On top of that, the gradient noise in a model with attention has heavy tails, with occasional batches producing enormous gradients, and the scale of the gradients differs by orders of magnitude between the embeddings, the attention matrices and the normalisations. They're ravines inside ravines, and SGD has one $\eta$ for all of them.

Each of the numbers in the table answers to a failure with a name.

$\beta_2 = 0.95$ rather than the original paper's $0.999$. The memory of $\mathbf{v}$ is $1/(1 - \beta_2)$ steps: a thousand with $0.999$, twenty with $0.95$. With a thousand steps of memory, when a batch suddenly brings a large gradient in some coordinate, $\mathbf{v}$ takes hundreds of steps to notice, and meanwhile $\hat{\mathbf{m}}/\sqrt{\hat{\mathbf{v}}}$ is far above $1$ in that coordinate: an enormous step, and the loss jumps. Those are the loss spikes that keep everyone who trains at this scale awake. With twenty steps of memory, $\mathbf{v}$ reacts sooner. GPT-3 set $0.95$, and nobody has had a reason to go back.

Gradient clipping to norm $1$. Before $\mathbf{g}$ enters $\mathbf{m}$ and $\mathbf{v}$, if its norm exceeds $1$, it gets rescaled to $1$. It's the other defence against the heavy tails: an atypical batch can't push the averages around.

$\varepsilon$. In the original paper it's $10^{-8}$; Llama 2 used $10^{-5}$, and OLMo 2 went back to $10^{-8}$ because with $10^{-5}$, the parameters whose gradients are smaller than that, which in a large model are many, see their step held back by $\varepsilon$ instead of by $\sqrt{\hat{\mathbf{v}}}$. A number that exists to avoid a division by zero ends up deciding how much the quietest parameters learn.

Warmup. The bias correction fixes the mean of $\hat{\mathbf{v}}$ in the first steps, not its variance: estimated from three or four gradients, $\hat{\mathbf{v}}$ is so noisy that the quotient can take large steps in the wrong directions. The RAdam analysis showed that raising the rate linearly from zero over the first few thousand steps is precisely a cure for that. Llama 2 warms up for $2\,000$ steps; Llama 3, for $8\,000$.

Schedule. After the warmup, comes a cosine decay down to $10$% of the peak rate in GPT-3 and Llama, or a long constant stretch followed by a cooldown in DeepSeek-V3 and Kimi K2, which hold the rate fixed for the first ten trillion tokens and only then bring it down. The two shapes give equivalent results, and both are Robbins and Monro's second condition made cheap: stay at the highest rate the training run can stand for most of the time, and lower it at the end to settle the trembling.

Then there's the bill. $\mathbf{m}$ and $\mathbf{v}$ are two vectors the size of the model, and for $405$ billion parameters in single precision they take $3.2$ TB: forty $80$ GB H100s dedicated to remembering. That bill is where the last section begins.

What comes next: per matrix, not per coordinate

Adam costs two things, memory and a diagonal approximation, and what has been proposed since attacks one or the other.

Less memory. Adafactor, from 2018, doesn't store the whole $\mathbf{v}$ for an $m \times n$ weight matrix: it stores the row sums and the column sums, $m + n$ numbers instead of $mn$, and rebuilds the matrix as their product. T5 was trained that way. 8-bit Adam stores $\mathbf{m}$ and $\mathbf{v}$ in quantised form. Lion, from 2023, found through an automatic search over programs, keeps only momentum and uses as its step the sign of a blend of the momentum and the gradient: one state vector instead of two, and the sign is the extreme case of Adam's rescaling, where every step has size exactly $\eta$.

More curvature. This is the approach that has changed the table. Adam treats the $mn$ numbers of a weight matrix as $mn$ parameters with no relation to each other, and they aren't unrelated: the matrix multiplies a vector, and its gradient $\mathbf{G}$ is a matrix too, with rows and columns that mean something. Shampoo, also from 2018, preconditions with two small matrices instead of a diagonal, one on each side:

$\mathbf{L}$ is $m \times m$ and $\mathbf{R}$ is $n \times n$: for a square matrix, that's twice Adam's memory rather than its square, and in exchange you get an approximation of $H^{-1}$ that is no longer diagonal. It took six years to make it run at scale, and in 2024 it won the AlgoPerf competition, training $28$% faster than the reference, an AdamW with Nesterov momentum. SOAP runs Adam inside the basis that Shampoo computes.

Muon does something more radical with the same premise. Instead of preconditioning the gradient, it takes the momentum matrix $\mathbf{M}_t$ of each weight matrix and replaces it with the nearest orthogonal matrix: if $\mathbf{M}_t = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{\top}$ is its singular value decomposition, the step is

the same matrix with all its singular values set to $1$. Computing that decomposition would be expensive, so the December 2024 blog postarticleMuon: An optimizer for hidden layers in neural networksKeller Jordan, 2024 · kellerjordan.github.ioWhere Muon was born, as a blog post rather than a paper. It explains the orthogonalisation, why it's done with Newton-Schulz rather than an SVD, and why it applies only to the matrices of the hidden layers. that proposed Muon approximates it with five iterations of a polynomial in $\mathbf{M}\mathbf{M}^{\top}$, using only matrix products, in low precision. Compare with what came before: Lion normalises each number, Adam normalises each coordinate by its recent magnitude, and Muon normalises each matrix by its spectrum. It's the same rescaling, one level up. And the argument for why it helps is the ravine once more, inside each matrix: the gradient of a transformer layer is nearly low-rank, with a few directions dominating it, and orthogonalising it gives the rare directions the same step size as the dominant ones. One reading puts all three in the same formula: each is steepest descent under a different norm, and choosing the optimiser means choosing what "a step of size $\eta$" means.

Muon exists only for matrices. The embeddings, the output layer, the biases and the normalisation weights stay on AdamW in every model that uses it, which is why the last three rows of the table are, strictly, Muon and AdamW at once. What took Muon from an experiment to a production modelpaperMuon is Scalable for LLM TrainingLiu, Su, Li et al. (Moonshot AI), 2025 · arXiv:2502.16982The two additions that took Muon from a 124-million-parameter experiment to a production model: weight decay, and the rescaling of the update so that AdamW's hyperparameters can be reused. were two additions from February 2025: weight decay, and a rescaling of the step so that AdamW's $\eta$ and $\lambda$ carry over without a new search. With those, Moonshot trained a $16$-billion-parameter model on $5.7$ trillion tokens and reported twice AdamW's compute efficiency. Five months later came Kimi K2: a trillion parameters, $15.5$ trillion tokens and, by its own report, not a single loss spike. Then came GLM-5 and DeepSeek-V4, which justifies the choice in two words: convergence and stability.

No schedule. The other line of work doesn't touch the direction of the step but the rest of the recipe. Schedule-Free AdamW replaces the rate schedule with an average of the points the trajectory passes through, and it won the no-hyperparameter-tuning track of the same AlgoPerf. It's the only one of these proposals that removes a number instead of adding an idea.

It's worth ending with a fair measurement. The most careful comparison to datepaperFantastic Pretraining Optimizers and Where to Find ThemWen, Ahn, Liu et al., 2025 · arXiv:2509.02046The fair comparison: eleven optimisers, each with its learning rate tuned. The gains over AdamW are real, they belong to the matrix methods rather than the scalar ones, and they shrink as the model grows., covering eleven optimisers each with its learning rate tuned, at sizes from $0.1$ to $1.2$ billion parameters, found that the announced gains of $1.4$ to $2$ times came partly from comparing against a badly tuned AdamW; that the real gains exist and all belong to the methods that precondition per matrix, Muon, SOAP and their relatives; and that they're worth $1.4$ times on the smallest model and $1.1$ on the largest, shrinking as the model grows. $10$% of the compute of a frontier training run is still a lot of money, and the stability DeepSeek cites isn't measured in that table. But the picture is more modest than the announcements and firmer than the scepticism: the gains belong to the matrices, and they're the ones that last.

What to take away

If you take away one thing, let it be this: the optimiser of language models wasn't chosen for being good; it was chosen for being affordable. Newton had the curvature of every direction and solved the problem in one step; at this scale it fits nowhere and doesn't survive the noise. Everything that has happened since Cauchy has been about buying back, on credit, what Newton had for free: momentum averaging over time, RMSProp rescaling per coordinate, Adam combining both, AdamW fixing what the rescaling had broken, and Muon rescaling per matrix. The gradient descent rule hasn't changed by a letter. What changes, each time, is what a step can cost.

And if you take away three more, let them be practical.

The defaults aren't arbitrary, and knowing what each one answers to tells you when to touch it. $\beta_2 = 0.95$ guards against spikes; clipping to $1$, against atypical batches; the warmup, against the variance of $\mathbf{v}$ at the start; the rate decay is Robbins and Monro's condition; the $0.1$ of weight decay is the regularisation that Adam without the W applied backwards. If you're training something small and calm, the original $0.999$ for $\beta_2$ is reasonable; if you see spikes, you know which number to look at first.

The learning rate is still the number that matters. In Adam it comes in units of the parameter, so $3 \cdot 10^{-4}$ means that, almost always, no weight moves more than that per step, and that meaning doesn't depend on the model. Before trying another optimiser, sweep $\eta$: a good part of the gains announced in recent years is explained by an AdamW whose rate nobody had swept.

If you're fine-tuning a model that isn't yours, the optimiser's memory decides whether it fits. $\mathbf{m}$ and $\mathbf{v}$ take twice the room of the weights, and every answer you've seen to that, LoRA, 8-bit Adam, Adafactor, is a way of not paying the bill from the Adam section. And if what you're training is mostly matrices, Muon for the matrices and AdamW for the rest is, today, what three of the largest models ever published do.

Back to the table. Four labs wrote the same paragraph because they arrived at the same price. Three have rewritten it because they found a way to pay for a little more of what Newton had for free: not the curvature of each coordinate, but that of each matrix. Gradient descent is still subtracting the gradient. What has changed, once again, is what fits in a step.