Technical deep dive

How PSSA works, in full.

Every mechanism below is taken from the source and the published checkpoints: a selective diagonal state-space recurrence, a 512-slot episodic memory with hyperbolic keys, plasticity through closed-form consolidation, and a gradient harness that checks every kernel against a scalar reference on every commit.

Numbers on this page come from the measured checkpoint chain and the benchmark runs. Where a value is schematic (for example the softmax mix shown on the main page) it is labeled as such.

01 · Specification

The whole model, measured.

A single PSSA block trains end to end. Parameter counts are read from the checkpoint header, not estimated.

Trainable parameters	1,544,704 (matched transformer baseline: 1,541,120)
Vocabulary	2,048 byte-pair tokens
Latent width	256 channels
State size	16 states per channel (256 × 16 values persist between tokens)
Time constants	16, log-spaced, horizons from 1.5 to 200 tokens
Episodic memory	512 slots, 32-d keys, 256-d values, fixed capacity
Plastic adapter	rank 16, updated every token; consolidated by a closed-form ridge step
ML framework	none. The linear algebra is written directly; gradients are checked numerically.

Checkpoint header: model=pssa parameters=1544704 vocab=2048, read from the published 64-checkpoint chain.

02 · Episodic memory

Keys on a hyperbolic ball, values in flat space.

Each of the 512 slots holds a key on the Poincaré ball and a value in ordinary Euclidean space. Distance on the ball grows sharply near the boundary, so broadly reused context sits near the origin and one-off episodes sit near the edge; a single distance function separates them without any learned metric.

d_H(u, v) = arccosh(1 + 2 ‖u − v‖²(1 − ‖u‖²)(1 − ‖v‖²) ) (1)

d_H — hyperbolic distance on the Poincaré ball.

‖·‖ — Euclidean norm of the slot key.

arccosh — inverse hyperbolic cosine; real-valued on [1, ∞).

The read is an exact scan. A query is formed from the current token embedding and the recurrent state, every occupied key is scored, and a softmax over negative distance turns the scores into a mix:

q_h = proj(W_qx x_t + W_qh h_t) (2)

w_s = softmax(− d_H(q_h, k_s)τ_mem) , m = Σ_s w_s v_s (3)

q_h — state-conditioned query, projected to the ball.

k_s, v_s — key and value of slot s.

τ_mem — temperature; smaller values sharpen the mix.

m — the retrieved value, blended into the block output.

The write path is round-robin by design: a pointer advances through the 512 slots one per token, and two gates decide whether the target slot may actually be overwritten. A novelty gate blocks writes that carry nothing new, and a refractory counter protects a recently written slot from immediate erasure. Capacity is therefore fixed and forgetting is explicit, not a side effect of a growing cache.

An optional flag, --memory-top-k K, still scans all 512 keys but mixes only the K best values. It exists for speed experiments; exact full-scan read is the default and the benchmarked path.

03 · Selective recurrence

One state, updated in place.

The core is a selective diagonal state-space recurrence in the Mamba family: the transition is diagonal, the step size and the input map are functions of the current token, and nothing outside the state is consulted.

δ_t = softplus(W_δ x_t) , A_t = exp(δ_t A) , B_t = δ_t B (4)

h_t = A_t ⊙ h_t−1 + B_t ⊙ x_t (5)

x_t — the embedded token, 256 channels.

h_t — the state, 256 × 16, carried across tokens.

δ_t — per-channel step size, selecting how far the token reaches back.

⊙ — elementwise product over the 16 states per channel.

Sixteen log-spaced time constants give the state horizons from 1.5 to 200 tokens before training moves anything, so short-range spelling and long-range topic structure coexist from initialization. Because A is diagonal and the recurrence is written scalar-first, every batched and parallel kernel has a plain reference implementation to be checked against.

04 · Plasticity

Two speeds of change, one closed-form fold.

A per-token gate decides how much of the memory read is admitted, and a rank-16 adapter carries the fast path:

g = sigmoid(w_gate · x_t) , A(x_t) = W₂ silu(W₁ x_t) (6)

At consolidation boundaries, the statistics the adapter accumulated are folded back into the base transition matrix by a ridge solve:

A_base ← A_base + (H^TH + λI)⁻¹ H^T ΔH (7)

H — the accumulated input statistics since the last boundary.

ΔH — what the fast path changed for those inputs.

λ — the ridge penalty, keeping the solve well-conditioned.

(H^TH + λI)⁻¹H^T — the closed-form ridge solution; no optimizer state, no learning-rate schedule.

The result is a model whose reading changes what its next read retrieves: memory writes are gated by novelty and refractory counters, and the fast adapter shifts behavior between consolidation points. The scope is honest: this is measured on a 12.7M-token run, not claimed as a general training recipe.

05 · Verification

Every shortcut has a reference.

A scalar reference path computes each gradient by hand, one element at a time. Every batched and parallel kernel is finite-differenced against it on every commit, and the harness fails the build on disagreement.

max | ∂_batched − ∂_scalar | ≈ 2.98 × 10⁻⁸ (8)

The measured maximum disagreement across the kernel set is 2.98 × 10⁻⁸, within the 3 × 10⁻⁸ gate. Finite-difference spot checks run alongside. This is why the results below can be quoted as model behavior rather than as artifacts of a fast-but-wrong kernel.

06 · Training setup

One epoch, 64 checkpoints, a matched baseline.

PSSA trains on a 12.7M-token WikiText-103 slice (64 links, roughly 199k tokens per checkpoint) on a single Tesla T4, in a 64-checkpoint chain. The baseline is a transformer trained on the same corpus at matched parameters (1,541,120 vs 1,544,704), same tokenizer, same held-out slice of 198,939 unseen tokens.

Training cross-entropy per checkpoint, from the published checkpoint chain. Hover a point for the exact value.

Inspect the sampled data table

Checkpoint	Loss	Tokens	Updates
1	5.733	199059	451
5	4.617	198983	458
9	4.388	198975	456
13	4.279	199101	447
17	4.208	198989	457
21	4.168	199038	452
25	4.221	198851	471
29	4.076	198986	454
33	4.028	198844	468
37	3.960	199093	451
41	3.970	198997	460
45	3.953	198966	458
49	3.907	198905	465
53	3.897	198963	460
57	3.936	198863	469
61	3.928	198986	455

07 · Results

Measured, on the same corpus, at matched parameters.

Final checkpoints, 1.54M parameters each. Bars animate on scroll; lower is better for cross-entropy and perplexity, higher for generation speed.

Metric	PSSA	Transformer
Held-out cross-entropy	3.997	4.429
Held-out perplexity	54.4	83.8
Generation speed, same CPU	~12×	1×
Training cross-entropy	3.98	4.43

Numbers are from the final checkpoints of each run, scored on the identical held-out slice of 198,939 tokens.

08 · Scope and limits

What this is, and is not.

PSSA is a 1.54M-parameter research prototype, trained once on a small corpus. It is not a general-purpose model, and the benchmarks compare it to one baseline at matched size, not to frontier systems. The claims that are load-bearing are the ones that are checkable: exact full-scan memory reads, a diagonal recurrence written scalar-first, closed-form consolidation, and a gradient harness with a measured 2.98 × 10⁻⁸ maximum disagreement. One seed, one run: the checkpoint chain is published so every number above can be re-derived.