things I think about, written down
How LLMs went from GPT-3's 2,048 tokens to 1M+ — positional encoding, attention rebuilt around sparse and linear layers, training data, optimizers, long-context RL, and the evals that separate advertised from effective context. With figures.
Weight-tied depth from Universal Transformers to Huginn and Ouro. Looping buys reasoning rather than knowledge, a recurrence is worth about the square root of a fresh block, the test-time depth dial stops turning at the depth you trained at — and the KV cache is where the bill actually lands. Four interactive figures and a census of every trained looped model.
Can you see loop training in a reasoning chain? 18 runs of Kimi K3 vs K2.6 across one-shot, bare continuation, and a Ralph-style stop-hook loop: a hidden ~86-token harness template, a hallucinated file workspace, opposite reasoning-budget curves — and a Busted section for the three findings my own audit killed. Five figures.
Put five frontier models in a bare shell — one bash command per turn — and they split into dreamers, workers, pragmatists, and grinders. Early data from a benchmark I'm building: confabulated completions, verification habits, long-range recall, with interactive charts.
A step-by-step walk through what a coding agent actually does between your prompt and its reply — context assembly, streaming, mid-flight tool execution, compaction, and the decision to loop again. Adapted from an interactive demo.