Gradient Dissent prep · Thu 22 Oct 2026 (week 43) · looped transformers

One Model to Loop Them All

NVIDIA's Elastic Latent Recurrence (Ladhak, Acharya, Akle Serano, Sun, Ginsburg, Puvvada; arXiv 2610.12582, 8 Oct 2026). It is a recipe for a looped language model whose one-loop setting is as good as an ordinary model of the same size, so one checkpoint can be served cheap or deep. Every extra pass through the shared middle block buys accuracy, and nothing is lost at one pass. The post-trained result, Koi-4B, scores an estimated 18 on the Artificial Analysis Intelligence Index at one loop and 29 at four.

−0.5 / +1.0

one loop vs. matched baseline

10-task average, 4B dense / 30B-A3B MoE hybrids

+7.3 / +4.8

points from 1 to 4 loops

same checkpoints; most of it at loop 2

1.16×

training compute vs. baseline

mixed depths only in the last 25%

18 → 29

Koi-4B, AA-Index estimate

loops 1 → 4, their own harness

−33%

output tokens at 4 loops

deeper passes, shorter answers

1×

KV memory at any loop count

last-loop cache; 2.5× at 4 loops with per-loop caches

The idea in one paragraph

Split the layers into an input block, a shared recurrent block (about the middle half) and an output block. At one loop every layer runs once, so the model costs exactly what an ordinary model of that size costs. At k loops the middle block runs k times, re-reading the input-block output each time. Earlier looped models paid a one-loop tax: at one pass they were well below a normal model of their size. The paper's example is Huginn, at 34.9 against 46.4 on ARC-Easy after 800B tokens. So you needed a separate model for the latency-sensitive tier. ELR removes the tax with three choices:

Input14 layers, run once

Recurrent ×k24 layers, shared, gated update

Output14 layers, run once

1 · Adaptive-delta update

The state moves a learned, per-token, per-channel step toward the block's proposal, s ← s + β(r − s), instead of being replaced or summed. Only this rule matches the baseline at one loop.

2 · Mix loop counts inside each update

A rotating 10–15% of data-parallel ranks always run one loop, so one-loop and deep gradients land in the same optimizer step. Same loop distribution sampled per step instead: the tax comes back.

3 · Tell each pass where it is

FiLM/AdaLN-style conditioning on progress i/k and budget k/K, zero-initialized and grafted on when mixed-depth training starts. It lowers loss at every extra loop.

Plus three cost cutters: train at one loop for the first 75% of tokens and mix depths only at the end (1.16× the baseline's training compute, against 1.62× for mixing throughout); in MoE models, reuse the first loop's expert routes on later loops (11.7% faster updates); and at inference keep only the last loop's attention KV cache (memory stays at the one-loop size).

What it brings over the earlier looped models

Prior work	What it did	What ELR changes
Huginn (Geiping et al., 2025)	Elastic depth by sampling one deep recurrence per training step; large one-loop tax	Mixes depths within a step and gates the update: parity at one loop
Ouro (Zhu et al., 2025)	Loops the whole stack, no input re-injection	Loops only the middle half, so one loop is an ordinary model
Parcae (Prairie et al., 2026)	Stability conditions and scaling laws; below its baseline at one loop, above at eight	Takes the ordinary model as the bar to clear at one loop, not compute-optimal looping
LoopFormer (Jeddi et al., 2026)	Conditions the shared stack on time and step size	Same signal, grafted late with zero init, so it starts as the one-loop checkpoint
RINS (Alabdulmohsin & Zhai, 2025)	One-recursion parity at 600M, with depth-specific adapters, in perplexity	No depth-specific parameters; downstream evals at 4B dense and 30B-A3B MoE
Retrofitted recurrence; Relaxed Recursive Transformers (2025)	Convert a trained non-recurrent model, with a curriculum or per-depth LoRA	Recurrent by construction; only the loop-count distribution changes late in training
Mixture-of-Recursions (Bae et al., 2025)	Per-token recursion depth	One loop count per request (simpler to serve; adaptive depth left open)

Prior-work characterizations follow the paper's own related-work section, which is where the Huginn and Parcae numbers come from.

The real contribution is the framing plus the training trick. Elasticity becomes a free option on a model you would have trained anyway, at about +16% training compute, and the within-step mixing is what makes it free. The systems pieces are what make it deployable: matched latency at one loop, route replay, last-loop KV.

Koi-4B: one checkpoint, four price points

Koi-4B, 1 loop

Koi-4B, 2 loops

Koi-4B, 3 loops

Koi-4B, 4 loops

Qwen3.5-4B

Qwen3.5-9B

Nemotron-3.5-Lightning30B-A3B

Gemma 4 26B-A4B

Qwen3.6-35B-A3B

Artificial Analysis Intelligence Index v4.1 estimates from the paper's own harness (Figure 3, Table 10), not Artificial Analysis's figures; CritPt is not run and counts as 0.

The second loop carries 8 of the 11 points. At two loops a 4B model sits level with Gemma 4 26B-A4B, and at four it trails only Qwen3.6-35B-A3B. Decode is 1.36× slower per token at two loops, but answers are 22% shorter, so a whole answer takes about 6% longer. At four loops it is 2.06× per token and about 1.4× per answer.

Strengths

Matched baselines at two scales. 4B dense hybrid on 1T tokens and 30B-A3B MoE on 300B tokens, same data and token budget as their non-recurrent twins. Matched baselines are rare in the looped-model literature.
Ablations that isolate each choice on a 0.5B model (300B tokens): state-update rules, fixed vs mixed depth, sampling schemes, override fraction, switch point, conditioning.
Cheap to adopt. No optimizer or learning-rate changes, late switch at about +16% training compute, about 0.2% extra parameters without conditioning.
Serving is taken seriously. One-loop latency equals the baseline's (measured in vLLM), and they show memory and latency curves, route replay and last-loop KV.
Carried through post-training to agentic benchmarks (Terminal-Bench, τ³-Bench, GDPval), where the extra loops also shorten outputs.

Weak spots to push on

"Matched" is matched in tokens, not compute

Parity at one loop costs +16% training FLOPs (+31% for Koi). And four loops cost 2.06× decode and 2.33× time to first token, with no comparison against an ordinary model given the same inference compute, say an 8B trained on the same data. The paper says that is a different question. It is still the one a deployer asks.

Koi has no matched baseline

Its "18 = Qwen3.5-4B" compares models trained on different data. So whether one-loop parity survives post-training is not tested. Every score is from their own harness, the Index is an estimate, and RL routed harder prompts to higher loop counts.

Parity is inside the noise

One training run per arm, no seeds. At one loop the 4B averages move −0.52 overall but −5.3 on GSM8K and +5.2 on MATH-500. "Within a point of the baseline" is as precise as the claim can be.

The elastic range is mostly one step

Loop 2 holds 79% (4B) and 87% (30B) of the 1→4 gain; K is only 4, and extrapolation past it is not tested. Their explanation, that later loops see fewer training examples, is not tested either.

Conditioning is validated at 0.5B only

The 4B and 30B matched runs have no conditioning; Koi has it but no ablation. Its gain at scale is inferred.

Last-loop KV is not free

−3.0 points on AA-LCR at four loops, and GDPval failures, mostly timeouts, rise from 12 to 52 of 220 tasks. Fewer output tokens come partly from fewer answers hitting the cap.

What is surprising

The gate helps with no recurrence at all. An ELR model trained only at one loop beats the plain baseline (validation loss 1.660 against 1.685 at 0.5B). So part of "no tax" may be an architecture improvement, and the baseline is not the strongest one-loop model.
How depths are grouped matters, not just how often each is drawn. With the same distribution over loop counts, putting several depths in one optimizer update lowers loss at every k. Sampling one depth per step at the same one-loop probability stays above the baseline.
Late beats early. Switching to mixed depth after 75% of tokens, with zero-initialized conditioning, beats mixed depth from scratch (1.6147 against 1.6273 at five loops) at a quarter of the extra cost.
Thinking deeper shortens the thinking. More loops, 22–33% fewer output tokens: latent depth substitutes for chain-of-thought tokens.
Experts drift, replay doesn't care. By loop 4 a token keeps only 53% of its first-loop experts, yet reusing the loop-1 routes costs +0.004 loss and nothing measurable downstream.
Fixed-depth looping is brittle. A model trained at one loop and run at five reaches loss 4.24 against 1.66 at its own depth. Elasticity has to be trained in.
Input re-injection, standard since Huginn, did not help at 0.5B (slightly worse at every loop count).

An outside check: Dory, the public sibling

Koi-4B is not on Hugging Face (checked Tue 20 Oct). Eight days after the paper, NVIDIA released Dory1.0-4B, announced by Boris Ginsburg, one of the paper's authors. It uses the same input/recurrent/output split with input re-injection and last-loop KV, on a different backbone: a normalized-GPT (nGPT), 8/20/8 blocks, 6T tokens with the first ~4.5T in NVFP4. We re-measured Dory against its card this week (full write-up). What carries over to this paper:

Benchmark	Model	1 loop	2 loops	4 loops
GPQA Diamond	Koi-4B (paper)	59.9	69.9	72.8
	Dory card	69.19	72.41	72.16
	Dory, our measurement	69.2	74.1	73.1
IFBench	Koi-4B (paper)	61.2	70.1	71.8
	Dory card	74.29	77.94	79.0
	Dory, our measurement	74.2	78.6	78.8
AA-LCR	Koi-4B (paper)	51.3	65.7	68.3
	Dory card	63	71	73
	Dory, our measurement	62.7	71.3	72.0

Three different harnesses (the paper's, Dory's card, ours), so compare shapes, not single digits. Our runs: vLLM on Modal, many repeats per loop count, NVIDIA's IFBench metric.

Diminishing returns hold up independently. On Dory the second loop is worth about +5 on GPQA, +4 on IFBench and +9 on AA-LCR. The 2→4 step is −1.0 ± 1.0, +0.2 ± 0.8 and +0.7 ± 2.7: nothing these sets can resolve. On GPQA one 2-loop pass matches a majority vote over five or six 1-loop samples at about a quarter of the compute.
Last-loop KV looked free on Dory. GPQA and IFBench differences are null at 2 and 4 loops, and at 4 loops it serves 1.34× the tokens per second on an H200 because the smaller cache fits twice the batch. Koi shows small losses, the largest on AA-LCR.
Shorter answers, but less so on single questions. Dory's answers are 11–18% shorter at 4 loops (GPQA, IFBench, AA-LCR). Koi's 33% is a geometric mean over agentic, multi-turn tasks; on its own GPQA it is 18%, on AA-LCR 44%.
The striking contrast. Dory starts 9–13 points above Koi at one loop, and the gap shrinks with every loop: 9.3 → −0.6 points on GPQA, 13.1 → 7.2 on IFBench, 11.7 → 4.7 on AA-LCR at four loops. How much a loop is worth seems to depend on how strong the one-loop model already is. A baseline-matched paper cannot show that; two checkpoints from one company can, with the caveat that the harnesses differ.

Questions for the table

At equal inference compute, does a looped 4B at 2 loops beat a dense model of about 5.5B, the same 1.36× cost, trained on the same data?
Why does loop 2 carry 80–90% of the gain: training exposure (deeper loops appear in fewer examples), or a fixed point in what the block can refine? Reweighting the loop distribution is the cheap test the authors name.
Is the one-loop "parity" really the gate acting as a learned skip connection? The fixed-1-loop result suggests a stronger one-loop baseline exists.
Why would grouping depths within one update matter? Gradient interference, an implicit consistency loss between depths, or optimizer statistics?
Does budget conditioning block per-token adaptive depth and extrapolation past K=4, the things Huginn and Mixture-of-Recursions are built for?
Koi vs Dory: what makes Dory's one-loop endpoint so much stronger, the nGPT backbone or 6T tokens of NVFP4 pretraining (vs Koi's 7T)? Post-training?

Paper: arXiv 2610.12582v1, 26 pages, read in full; numbers are from its text, Tables 1–14 and Figures 2–7. Dory: model card (revision b366b87f) and our re-measurement, Dory1.0-4B, re-measured (Mon 19 – Tue 20 Oct 2026). Prepared Tue 20 Oct 2026 (week 43).