Vitasta: A Recurrent-Core Hybrid Architecture

Abstract

Large language models built on full self-attention share six standing limitations: compute that grows quadratically with context length, hard context limits, degraded use of long contexts, weak tracking of changing state, dependence on added positional signals for word order, and heavy data and compute demands. We present Vitasta, a language-model architecture designed to change how models handle all six. Its backbone is a gated delta-rule recurrent layer that keeps a fixed-size associative state, updates it token by token, and gets word order from the recurrence itself. A few recall layers, which combine a local attention window with retrieval from a bounded chunk memory, give exact access to distant text.

We contribute a stability result for the write range, an order-sensitivity identity for the state, a multi-timescale forgetting scheme, and a cost analysis that gives the best chunk size in closed form. Testing an early version of the write-strength gate found no benefit over a simpler input-dependent rule. Testing the semantic chunk router found it reliable for topically coherent text but unable to find an isolated, thematically unrelated fact; we add a second, exact-match retrieval channel to close this gap. This is a design and analysis report. Overall progress: 25.3% on a defined validation ladder.

1. Introduction

Transformers built on full self-attention underlie nearly all current large language models. Their success comes with six recurring limitations, which are the design targets of this work:

1. Quadratic cost. Self-attention compares every token with every other, so compute grows with the square of the context length n.
2. Context limits. Models are trained at a fixed maximum length and degrade beyond it, and the key-value cache grows linearly with n.
3. Context rot. Even inside the window, the use of information degrades as the context grows and depends on where the information sits.
4. Poor dynamic state tracking. Tasks whose answer depends on an evolving state — counters, variables, permutations — are hard for fixed-depth attention.
5. No inherent word order. Attention is order-blind and needs added positional signals such as rotary embeddings, which can limit length extrapolation.
6. Data and resource hunger. Compute-optimal training of Transformers calls for token budgets that grow with model size.

2. Model Architecture

2.1 Overview

Vitasta is a stack of G identical groups. Each group contains k flow blocks followed by one recall block. A block has two sublayers — a mixer and a gated MLP — each with a residual connection and RMSNorm. The mixer is a flow layer in flow blocks and a recall layer in recall blocks. The input is a token embedding with no positional table.

2.2 Flow Layer

Each head of a flow layer keeps a matrix state S_t ∈ ℝ^d_v×d_k. Given unit-norm key and query k_t, q_t, and a value v_t, computed from the layer input x_t, the state is updated and read as:

Here α_t ∈ (0, 1] decays old content and β_t ∈ (0, 2) sets the write strength. The factor (I − β_tk_tk_t^⊤) first erases the part of the state along the key, and the last term writes the new pair. This is the gated delta-rule form with the write range of [11].

2.3 Stability and State Tracking

Proposition 1. Let ‖k_t‖ = 1, β_t ∈ [0, 2] and α_t ∈ (0, 1]. Then the update satisfies:

At β_t = 1 (and α_t = 1) the old value stored at k_t is replaced by v_t. At β_t = 2 it is replaced by 2v_t − S_t−1k_t, and with v_t = 0 its sign flips — enough to count parity. The upper end of the range is the stability limit: for β_t > 2 the factor |1 − β_t| exceeds 1 and Proposition 1 no longer holds.

2.4 Multi-Timescale Forgetting

Head h ∈ {0, …, H − 1} decays with:

Base rates are set so that the half-life at Δ = 1 grows geometrically across heads:

For H = 12, n_min = 16 and n_max = 65,536, half-lives run from 16 to 65,536 tokens and each head's is about 2.13× its predecessor's.

2.5 Order Without Positional Encodings

Order is not added to the input; it is a property of the state. Unrolling Eq. (1) gives:

Proposition 2. For any two tokens a and b, the commutator of P_a and P_b is:

It vanishes only when the keys are orthogonal or parallel, or a write strength is zero. Swapping two tokens whose keys partially overlap therefore changes the state — the layer is order-sensitive without any positional signal.

2.6 Write-Strength Gating — A Tested Simplification

An earlier version scaled write strength by surprise. We tested this in isolation on three synthetic streams: clean, noisy, and one with an abrupt distribution shift. The surprise-gated rule was never the best option. On the noisy stream, its MSE was roughly 5× higher than a constant write strength, because sensor noise looks identical to genuine surprise. We adopt the simpler rule:

2.7 Recall Layer and Chunk Memory

A recall layer is causal attention over the last W tokens plus a set of retrieved chunks. Tokens leaving the window are grouped into chunks of C tokens and stored with their keys, values, and one summary key:

A query scores the summaries:

and attends, with an ordinary softmax, to the tokens of the top-k scoring chunks together with the window.

2.8 Exact-Match Retrieval

Mean-key routing is semantic. Testing with controlled topical coherence found: 96–100% retrieval for coherent chunks, but chance (~16%) for isolated facts regardless of training. We add an exact-match channel:

In the isolated-fact case that broke the semantic router, the query placed 61% of its attention mass on the correct earlier occurrence, versus 0% for the router alone.

2.9 Training Objective

The main loss is next-token prediction. Auxiliary heads predict tokens 2 to m positions ahead:

The training context length follows a geometric curriculum from n₀ to n_max over s_w steps:

3. Complexity and Memory

Layer type	Compute (training)	Inference memory
Self-attention	O(n²·d)	O(n·d)
Flow layer	O(n·d·dh)	O(d·dh) — fixed
Recall layer	O(n·d·(W + n/C + kC))	O(M·C·d) capped

The per-token cost of a recall layer is the sum of three terms:

The recall cost is minimized at:

For n = 131,072 and k = 8, C* = 128. The flow state occupies H·d_k·d_v numbers per layer — for H = 16 and d_k = d_v = 64, that is 65,536 numbers or 128 KiB in bf16.

4. Validation Plan

1. Synthetic tasks. Multi-query associative recall, state tracking (permutations and parity), and copying. The model should still work at lengths several times longer than in training.
2. Small-scale scaling. Train three sizes (~125M, 350M and 1B parameters) and a same-compute Transformer baseline, and compare loss against FLOPs and wall-clock time.
3. Long context. Needle-in-a-haystack and RULER at several times the training length, looking for sudden drops.
4. Efficiency. Tokens per second and memory at 32k and 128k context, for training and inference.
5. Ablations. Remove recall layers; compare write strength up to 1 vs up to 2; remove chunk memory; vary flow-to-recall ratio; remove multi-token prediction.

5. Progress

Stage	Weight	Status
1 — Specification of mechanisms	15	Complete
2 — Analytical and numerical checks	15	Partly complete
3 — Reference implementation	15	Not started
4 — Synthetic-task validation	20	Not started
5 — Scaling against a Transformer baseline	20	Not started
6 — Long-context and efficiency evaluation	15	Not started

Overall progress: 25.3%. This number measures how far the validation ladder has been climbed. It is not a measure of model quality, and no problem counts as solved until stages 4–6 are complete.

6. Limitations and Risks

1. Exact recall from far back is the known weakness of recurrent cores. Recall layers are tested on controlled synthetic data with encouraging results but not on a real trained model yet.
2. The head spectrum is an untested proposal and may not help; the surprise gate it was paired with has already been tested and dropped.
3. The write range up to 2 and the chunk memory need adapted or custom kernels.
4. The architecture addresses data and resource hunger only in part.
5. Retrieval cost still grows with length (though sub-quadratically).

References

1. Vaswani et al. Attention is all you need. NeurIPS, 2017.
2. Liu et al. Lost in the middle. TACL, 2024.
3. Hsieh et al. RULER. arXiv:2404.06654, 2024.
4. Merrill et al. The illusion of state in state-space models. ICML, 2024.
5. Su et al. RoFormer. arXiv:2104.09864, 2021.
6. Kazemnejad et al. Impact of positional encoding on length generalization. NeurIPS, 2023.
7. Schlag et al. Linear transformers are secretly fast weight programmers. ICML, 2021.
8. Yang et al. Parallelizing linear transformers with the delta rule. NeurIPS, 2024.
9. Gu & Dao. Mamba. arXiv:2312.00752, 2023.
10. Yang et al. Gated delta networks. ICLR, 2025.
11. Grazzi et al. Unlocking state-tracking in linear RNNs. ICLR, 2025.
12. Lieber et al. Jamba. arXiv:2403.19887, 2024.
13. Jelassi et al. Repeat after me. ICML, 2024.
14. Arora et al. Zoology. ICLR, 2024.
15. Lu et al. MoBA. arXiv:2502.13189, 2025.
16. Behrouz et al. Titans. arXiv:2501.00663, 2024.
17. Gloeckle et al. Multi-token prediction. ICML, 2024.
18. Robertson & Zaragoza. BM25 and beyond. FnTIR, 2009.
19. Zhang & Sennrich. RMSNorm. NeurIPS, 2019.
20. Shazeer. GLU variants. arXiv:2002.05202, 2020.
21. Hoffmann et al. Compute-optimal LLMs. NeurIPS, 2022.