Tech · yifanzhang-pro.github.io
Recurrent Looped Transformer
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Known tokens can be encoded in a causal batch.
Key facts
- The concrete configuration uses 48 encoder layers and 48 decoder layers, with compatible attention and FFN weights shared across stages
- Each token executes 96 logical blocks; decoder cross-attention means these blocks do not all have equal FLOPs
- Recurrent Looped Transformer (RLT) combines a causal encoder with a recurrent decoder that carries its final hidden state and layerwise sliding-window attention (SWA) cache across every prompt
- Rebuild the full history under current parameters, including prompt states and decoder SWA KV
Summary
Recurrent Looped Transformer (RLT) combines a causal encoder with a recurrent decoder that carries its final hidden state and layerwise sliding-window attention (SWA) cache across every prompt and response token. The design brings together latent reasoning with unbounded temporal depth, model–hardware co-design, and model–RL algorithm co-design. Infinite depth refers to an extensible temporal path, not infinite work within a token. Rebuild the full history under current parameters, including prompt states and decoder SWA KV. Here \(M_{\le t}\) is global encoder memory, \(s_t\) is the recurrent output, and \(C_t^D\) contains layerwise decoder KV.