Blog
What I explain in lessons, written down
Articles on programming, mathematics and AI. One concept per article, with the hard part included rather than tucked out of sight.
Why AdamW for training language models, and what is changing
Of all the optimisation methods that exist, language models are trained with the simplest one, and with one specific variant of it: AdamW. Why the only information you can afford at that scale is the gradient, how every improvement since buys back on credit the curvature Newton had for free, and what has started to change in 2026.
Read the articleThe evolution of the tokeniser: from words to bytes
The Spanish sentence "El niño juega en el jardín" cost twelve tokens in 2019 and costs seven today, the same as its English translation. The story of why: from the word tokeniser to BPE, WordPiece, Unigram and the byte floor.
Read the articleSign in to be notified when a new article goes up