Looped transformers: what a recurrence is worth

September 6, 2026 · ~15 min read · Draft — figures first, prose still being trimmed

On weight-tied depth, the exchange rate between parameters and FLOPs, and why the loop turns out to be an architecture decision rather than an inference-time dial.

Here is a result that should be strange. Ouro-1.4B, a 1.4-billion-parameter base model from a ByteDance Seed-led collaboration, scores 82.4 on MATH500 against Qwen3-4B-Base’s 59.6 — a model with three times its parameters and five times its training tokens. On MMLU the same two models go the other way: 67.4 against 73.2 (Zhu et al., 2025, Table 7). It wins by 22.8 points on the maths benchmark and loses by 5.8 on the knowledge one.

The architecture behind that split is the simplest idea in the field: take a block of layers and run it more than once. Same weights, again. Drag the slider.

In code the whole idea is one for loop. This is the Huginn layout — a prelude computed once, a core applied r times with the input re-injected on every pass, a coda — with Block an ordinary pre-norm attention-plus-MLP layer. Small enough to run on a laptop, and it does — open the block to see it.

class LoopedLM(nn.Module):
    """Prelude -> core x r -> coda, the Huginn layout (Geiping et al., 2025)."""
    def __init__(self, vocab, d, heads=4, n_prelude=2, n_core=4, n_coda=2):
        super().__init__()
        self.embed = nn.Embedding(vocab, d)
        self.prelude = nn.ModuleList(Block(d, heads) for _ in range(n_prelude))
        self.adapter = nn.Linear(2 * d, d)      # A: concat(s, e) -> core input
        # the only weights the loop ever touches:
        self.core = nn.ModuleList(Block(d, heads) for _ in range(n_core))
        self.coda = nn.ModuleList(Block(d, heads) for _ in range(n_coda))
        self.head = nn.Linear(d, vocab)

    def run_prelude(self, tokens):
        T = tokens.shape[1]
        self.mask = torch.triu(torch.full((T, T), float("-inf")), diagonal=1)
        e = self.embed(tokens)
        for blk in self.prelude: e = blk(e, self.mask)
        return e                            # e = P(x), computed once per token

    def run_core(self, e, s):               # s_i = R(e, s_{i-1})
        h = self.adapter(torch.cat([s, e], dim=-1))   # input injected EVERY pass
        for blk in self.core: h = blk(h, self.mask)
        return h

    def run_coda(self, s):
        for blk in self.coda: s = blk(s, self.mask)
        return self.head(s)                 # p = C(s_r)

    def forward(self, tokens, r):
        e = self.run_prelude(tokens)
        s = torch.randn_like(e) * 0.4       # s_0 ~ N(0, 2/5): noise, not e
        for _ in range(r):
            s = self.run_core(e, s)
        return self.run_coda(s)

stored: 537K parameters, of which the core is 200K
at r = 32 the core runs 32 times: 6735K parameter-applications per token
logits: (2, 16, 1000)

Three choices in those lines carry the paper. The input e is concatenated back in at every iteration — without it the fixed point would depend only on the starting state, which is why the uninjected models in §2’s table diverge. The starting state is noise, not the embedding, so that different starts converge to the same trajectory (the “path independence” Huginn borrows from the deep-thinking line). And the core is the only weight matrix the loop touches — the block prints Figure 1 in numbers: 537K parameters stored, 6.7M applied per token. The real model adds a second norm after every sublayer; at 3.5B that turned out to be required.

And one thread I did not expect: the story most people absorbed from 2025 is that loops give you a dial — run more iterations at inference, get more accuracy, a third axis of test-time compute. That is the part the 2026 evidence is least kind to.

The Idea Is Eight Years Old and It Lost on the Merits

Weight-tied depth was tried, measured carefully, and set aside. The Universal Transformer (2018) had striking headline numbers — LAMBADA perplexity 142 against a vanilla Transformer’s 7,321 — but always at matched parameters. Matched compute told a different story, and by 2022 the Universal Transformer’s own co-author had run the comparison that closed the programme.

Tay et al.’s summary line: “amongst all ten architectures that we consider, the vanilla Transformer has the best scaling behaviour.” Hold on to the last row — a loop count is a depth the model was fitted to, not a knob. That footnote from 2018 becomes the main event in §7.

What Changed Was the Accounting, Not the Architecture

Nothing above was refuted. What changed is what the numbers are worth.

The second change is a training recipe that survives a billion-parameter run — three ingredients, the first two borrowed from the “deep thinking” line, none standard in pre-2024 looped transformers:

Two of the three ingredients are a dozen lines each. Draw the depth per sequence from a distribution with a heavy right tail, and cut the gradient before the last k iterations:

import math, torch

def sample_depth(r_bar=32, sigma=0.5):
    """Log-normal Poisson unrolling (Huginn): most draws below r_bar, a heavy tail."""
    tau = torch.normal(math.log(r_bar) - sigma ** 2 / 2, sigma, size=())
    return int(torch.poisson(tau.exp())) + 1

def looped_loss(model, tokens, targets, r, k=8):
    """Backprop through only the last k of r iterations; memory stops growing with r."""
    e = model.run_prelude(tokens)
    s = torch.randn_like(e) * 0.4
    with torch.no_grad():                   # iterations 1..r-k: forward only, no graph
        for _ in range(max(0, r - k)):
            s = model.run_core(e, s)
    for _ in range(min(r, k)):              # the last k keep their activations
        s = model.run_core(e, s)            # e still carries gradient into the prelude
    logits = model.run_coda(s)
    return torch.nn.functional.cross_entropy(logits.flatten(0, 1), targets.flatten())

sampled depths: [61, 27, 37, 43, 19, 48, 15, 9, 33, 60]
loss 7.05; gradient reaches prelude=True core=True coda=True

Ten draws at r̄ = 32 range from 9 to 61, and that spread is the whole reason Huginn tolerates depths it was not trained at (§7). The no_grad block is what makes r = 61 affordable: activations are kept for the last eight iterations only, so memory does not grow with depth. The prelude still gets a gradient — e is re-injected inside the graph — and this is exactly the shortcut the φ paper found costs capacity: truncation lowers the exchange rate from 0.45 to 0.38 (§5). Huginn additionally locks one r per micro-batch across workers, so no GPU waits on the deepest draw.

Every Trained Looped Model I Could Find — and Yes, Some Have Shipped

Before the analysis, the cast: every model I found that was actually trained with weight-tied depth. Two patterns. The research models loop many times and stop under 2B parameters. The ones that shipped with open weights — a 40B coder in December 2025, a 7B coder trained on 18T tokens and a 3B generalist trained on 28T tokens in mid-2026 — loop exactly twice, at a fixed depth, with a separate KV cache per pass. Nanbeige reports that more passes gave “only marginal additional improvement” and made optimisation less stable; LoopCoder-V2 found its third loop regressed. The practitioners landed on the same two loops the compute-matched studies in §5 point to.

“Stored → effective” is unique parameters against the depth or FLOPs the loop unrolls to. “Weights” means downloadable checkpoints, checked on 7 September 2026, not code. The last column is each paper’s own headline against its own baseline. Reports that OpenAI’s GPT-6 uses a constrained form of recurrent depth are unconfirmed — its system card says nothing about architecture — so it is not in the table.

Loops Buy Reasoning, Not Knowledge

Write (k ⊗ L) for a k-layer block looped L times. It has the parameters of (k ⊗ 1) and the FLOPs of (kL ⊗ 1), so you can ask exactly how much of the gap between the small model and the big one the loop recovers (Saunshi et al., 2025, 1B scale, the Pile).

One underrated caveat. Saunshi’s middle-looping variant — untied first and last block, loop the middle — gets closed-book QA back to 11.0 against the baseline’s 11.2, at the cost of most of the math-word-problem gain (34.3 → 28.3). Subformer (2021), Huginn, Mixture-of-Recursions and SMELT all keep the ends untied too. The knowledge penalty looks substantially like an artefact of tying everything, not a law about weight sharing.

A Recurrence Is Worth About the Square Root of a Fresh Block

In April 2026 Schwethelm, Rückert and Kaissis put a number on it, fitting a Chinchilla-style law with recurrence as an exponent on the looped parameters: L = E + A·(N_once + r^φ·N_rec)^(−α) + B·D^(−β). φ = 1 would mean a loop is worth a fresh block; φ = 0 would mean it is worth nothing. Over ~50× in training compute they measure φ = 0.46.

SMELT and the φ paper disagree, and nobody has run both protocols on the same models — SMELT’s point is that fixing FLOPs alone shrinks the looped model’s unique parameters, so φ may be measuring parameter loss. Where they agree is the shape: the margin is least unfavourable at the smallest loop counts and erodes as r grows. The most controlled win in the literature loops exactly twice.

A Loop Is Easy to Train Into a No-Op

The best section of the Huginn paper describes the two runs that failed. Two of three attempts at a large recurrent-depth run did not make the loop load-bearing — one collapsed, one left the recurrence decorative — and 2026 turned that folklore into mechanism.

The sting in the last row: those drifted models are nearly depth-invariant (perplexity moves +0.01 from one loop to four), so a calibrated halting rule exits at an average of 1.00 loops and reports a 3.4× speedup — a spectacular adaptive-compute number from a model that cannot use its loops at all. Carry that lens into every efficiency claim below.

The Test-Time Dial Turns About as Far as You Trained It

Ouro’s own depth ablation is the cleanest test: one checkpoint, evaluated at every loop count from 1 to 8, trained with depth capped at 4. Drag the cursor past the dashed line.

So loop count behaves like an architectural hyperparameter you commit to at training time. If you want flexibility at inference, you buy it by randomising depth during training, not by turning a knob afterwards — and “more context in, more depth worth spending” is a better argument for recurrence than “turn the dial up.”

Adaptive Depth Is Mostly a Threshold, and the KV Cache Is the Real Bill

If depth is not a free dial, the remaining pitch is adaptive computation: loop where it helps, exit where it does not. Two things get in the way.

The stopping rules in that table are each three lines, and Huginn’s needs no training at all. Read the logits out at every iteration and stop when they stop changing:

import torch, torch.nn.functional as F

def kl_exit(logits_prev, logits_new, thresh=5e-4):
    """Huginn, zero-shot: stop once consecutive iterations predict the same thing."""
    p, q = F.log_softmax(logits_prev, -1), F.log_softmax(logits_new, -1)
    return (q.exp() * (q - p)).sum(-1).mean().item() < thresh

def margin_exit(logits, thresh=2.0):
    """Popescu et al.: a logit-margin threshold beats the learned gates."""
    top2 = logits.topk(2, dim=-1).values
    return (top2[..., 0] - top2[..., 1]).mean().item() > thresh

def hidden_delta_exit(s_prev, s_new, eps=1e-2):
    """Ouro's baseline heuristic: the state stopped moving."""
    rel = (s_new - s_prev).norm(dim=-1) / s_prev.norm(dim=-1)
    return rel.mean().item() < eps

def generate_step(model, tokens, r_max=32):
    e = model.run_prelude(tokens)
    s = torch.randn_like(e) * 0.4
    prev = model.run_coda(s)
    for i in range(1, r_max + 1):
        s = model.run_core(e, s)
        logits = model.run_coda(s)          # read out at every iteration
        if kl_exit(prev, logits):           # or margin_exit / hidden_delta_exit
            return logits, i
        prev = logits
    return logits, r_max

KL-exit fired after 9 of 32 iterations

The block ends by running the rule on the toy model, and the line it prints is a warning, not a demo: KL-exit fired after 9 of 32 iterations. The model is untrained and its loop still drifts to a fixed point within nine iterations — the layer norms contract the state whether or not the iterations compute anything. A convergence test tells you the state stopped moving, not that the loop did work: the same trap that produced the 3.4× “speedup” from a depth-invariant model in §6.

Figure 5 is three lines of arithmetic — K and V, per layer, per head, per stored loop state:

def kv_bytes_per_token(layers, kv_heads, head_dim, loops=1, dtype_bytes=2):
    """K and V, per layer, per head, per stored loop state — bf16."""
    return 2 * layers * kv_heads * head_dim * dtype_bytes * loops

# layers, kv_heads, head_dim, stored loop states, weights in GB
models = {
    "Qwen3-1.7B":         (28, 8, 128, 1, 3.44),   # GQA: 8 KV heads
    "MELT-1.6B":          (24, 16, 128, 1, 3.27),  # one gated cache for all 4 loops
    "Ouro-1.4B-Thinking": (24, 16, 128, 4, 2.87),  # one cache per loop
}
ctx = 32_000                    # MELT reports decimal MB/GB at 32,000 tokens
for name, (L, H, D, loops, w_gb) in models.items():
    b = kv_bytes_per_token(L, H, D, loops)
    gb = b * ctx / 1e9
    line = f"{name:19s} {b/1e6:.3f} MB/token -> {gb:5.2f} GB at 32K"
    print(line + f", {w_gb + gb:4.1f} GB with weights")

Qwen3-1.7B          0.115 MB/token ->  3.67 GB at 32K,  7.1 GB with weights
MELT-1.6B           0.197 MB/token ->  6.29 GB at 32K,  9.6 GB with weights
Ouro-1.4B-Thinking  0.786 MB/token -> 25.17 GB at 32K, 28.0 GB with weights

Ouro stores 16 heads of 128 dimensions per layer with no grouped-query sharing and keeps four states, so 0.786 MB per token, the figure in MELT’s table; Qwen3-1.7B’s eight KV heads and one state come to 0.115. At 32,000 tokens that is 25 GB of cache against 3.7.

Does a Loop Go Brrr? Only Above the Saturation Batch

The question Horace He asks of any kernel applies to a decode step: are you bound by compute, by memory bandwidth, or by overhead? Count the bytes and the FLOPs. Every pass through the core re-reads its weights from HBM, and every stored KV state is read once per token generated:

BW, PEAK = 3.35e12, 990e12   # H100 SXM: HBM bytes/s, dense bf16 FLOP/s
RIDGE = PEAK / BW            # ~295 FLOP/byte: below this a step is memory-bound

def decode_step(once, loop, r, batch, ctx, kv_tok, d, layers):
    """One decode step for `batch` sequences; weights re-read from HBM every pass."""
    weight_bytes = 2 * (once + r * loop)             # bf16; the core is read r times
    kv_bytes = batch * ctx * kv_tok                  # every sequence's cache, once
    flops = batch * (2 * (once + r * loop)           # 2 FLOP per parameter-application
                     + 4 * layers * d * ctx)         # + attention over the context
    t = max((weight_bytes + kv_bytes) / BW, flops / PEAK)   # the roofline
    return 1e3 * t / batch                           # ms per token

CTX = 4096
# params applied once, params in the loop, r, KV bytes/token, d_model, layers
models = {
    "Huginn 3.5B, r = 32":       (2.0e9, 1.5e9, 32, 2.66e6, 5280, 132),  # 32 KV states
    "Ouro 1.4B, r = 4":          (0.0,   1.4e9,  4, 0.79e6, 2048,  96),  # 4 KV states
    "dense 50B (same FLOPs)":    (50e9,  0,      1, 0.33e6, 8192,  80),  # GQA, 8 KV heads
    "dense 3.5B (same weights)": (3.5e9, 0,      1, 0.13e6, 3072,  32),
}
B = (1, 8, 64, 512)
print(f"{'ms per token':28s}" + "".join(f"{'B=' + str(b):>9s}" for b in B))
for name, (once, loop, r, kv, d, L) in models.items():
    row = [decode_step(once, loop, r, b, CTX, kv, d, L) for b in B]
    print(f"{name:28s}" + "".join(f"{x:9.2f}" for x in row))

ms per token                      B=1      B=8     B=64    B=512
Huginn 3.5B, r = 32             33.10     6.98     3.72     3.31
Ouro 1.4B, r = 4                 4.31     1.38     1.02     0.97
dense 50B (same FLOPs)          30.25     4.13     0.87     0.46
dense 3.5B (same weights)        2.25     0.42     0.19     0.16

This is a model, not a measurement — an H100’s 3.35 TB/s and 990 TFLOP/s, bf16 weights, nothing overlapped — but it says three things the measured papers agree with. Below the ridge (about 295 FLOP per byte, so a few hundred tokens per step) a decode step is memory-bound and the loop turns one weight read into r of them: at batch 1 Huginn spends ~33 ms per token, more than the 50B dense model it matches in FLOPs and fifteen times more than the 3.5B dense model it matches in weights. At large batch the step should become compute-bound and cost what the FLOPs say — but a per-loop KV cache does not let it: with 32 stored states per token, Huginn’s cache traffic dominates and it flattens near 3.3 ms where the dense model reaches 0.5. Ouro flattens near 1 ms for the same reason; MELT’s single gated cache would put it near 0.25. So the memory a loop saves is weight capacity — Huginn trained with data parallelism alone — not bandwidth, and only a shared cache makes the loop bandwidth-neutral. The continuous-depth-batching paper measured exactly this shape: step time flat below a saturation batch, and the saturation batch falling with context length.

Loops Buy Serial Time; Only Tokens Buy Space

Giannou et al. hand-built a nine-layer looped transformer that runs a Turing-complete instruction set: “without this loop, the size of the transformer would have to scale with the number of lines of code.” Saunshi et al. proved a looped model can reproduce any m steps of chain of thought in m loops — given m dummy tokens in the context. That detail is the whole theory in one line.

Loops win where working state is bounded and steps are many — modular arithmetic, multi-hop composition, graph reachability — and do not win where the task needs an externalised scratchpad.

Scorecard

What would change my mind, in order:

• Randomised depth at Ouro’s scale. Huginn had the sampling and 800B tokens; Ouro had 7.7T tokens and a cap of 4. Nobody has run both with a 2026 stability recipe.
• Both compute-matching protocols on the same models. SMELT and φ = 0.46 disagree only about what is held fixed.
• Loops against chain of thought at matched inference FLOPs. The theory says loops simulate CoT up to a memory budget; I found no measured head-to-head.

The case for looping is a case about parameter efficiency — memory, footprint, bandwidth — with a real inductive bias toward multi-step computation. It is not a case about getting compute for free, and 2025 spent a year arguing it was — which is why the models that actually shipped loop twice, at a fixed depth, and stop there. If you take one number away: a recurrence is worth about the square root of a fresh block, and you pay full price in FLOPs either way.

References & Further Reading

1. The lineage. Graves, Adaptive Computation Time (2016); Dehghani et al., Universal Transformers (2018); Dabre & Fujita, Recurrent Stacking of Layers (2018); Bai, Kolter & Koltun, Deep Equilibrium Models (2019); Lan et al., ALBERT (2019); Reid et al., Subformer (2021); Takase & Kiyono, Lessons on Parameter Sharing (2021); Banino et al., PonderNet (2021); Tay et al., Scaling Laws vs Model Architectures (2022); Tan et al., Sparse Universal Transformer (2023); Mohtashami et al., CoTFormer (2023).
2. Theory. Giannou et al., Looped Transformers as Programmable Computers (2023); Yang et al., Looped Transformers are Better at Learning Learning Algorithms (2023); Back de Luca & Fountoulakis, Simulation of Graph Algorithms (2024); Fan et al., Looped Transformers for Length Generalization (2024); Xu & Sato, On Expressive Power of Looped Transformers (2024); Saunshi et al., Reasoning with Latent Thoughts (2025); Merrill & Sabharwal, A Little Depth Goes a Long Way (2025); Zhang, Chain-of-Thought and Compressed Looped Transformers (2026).
3. Recurrent-depth language models. Bae et al., Relaxed Recursive Transformers (2024); Geiping et al., Scaling up Test-Time Compute with Latent Reasoning (Huginn, 2025); Bae et al., Mixture-of-Recursions (2025); Zhu et al., Scaling Latent Reasoning via Looped Language Models (Ouro, 2025).
4. Scaling laws and stability, 2026. Prairie et al., Parcae; Schwethelm, Rückert & Kaissis, How Much Is One Recurrence Worth?; Sharma & Vu, The Readout Blind Spot; Wang et al., On the Residual Scaling of Looped Transformers; Kuo et al., Stabilizing Extrapolation via Learned Stochastic Stopping; Wang et al., SMELT.
5. Serving and adaptive depth. Popescu et al., Diagnosing Learned Halting Gates (2026); Vendrell et al., MELT (2026); Schwethelm et al., Continuous Depth Batching (2026); Mozer et al., Recirculation (Google DeepMind, 2026).
6. Latent reasoning, interpretability, safety. Hao et al., Coconut (2024); Snell et al., Scaling LLM Test-Time Compute Optimally (2024); Korbak et al., Chain of Thought Monitorability (2025); Weng, Why We Think (2025); Blayney et al., A Mechanistic Analysis of Looped Reasoning LMs (2026); Dilgren & Wiegreffe, Are Latent Reasoning Models Easily Interpretable? (2026); Wang & Reid, Looped Transformers under the Jacobian Lens (2026).

Sources were gathered with a fan-out of research agents and every number was checked against the paper it is attributed to; where a claim rests on an abstract or a single small-scale sweep, the tables say so. If you spot an error — especially in the 2026 citations — I would genuinely like to know.