PSSA is a 1.54M-parameter language model that is not a transformer. A fixed-size recurrent state carries the context. A bounded episodic memory looks it up. Closed-form consolidation folds experience back into the weights.
$ pssa --checkpoint 64 --report model pssa parameters 1,544,704 baseline 1,541,120 vocabulary 2,048 bpe latent width 256 state 256 × 16 · 16 log-spaced time constants memory 512 slots · 32-d keys · 256-d values framework none · linear algebra written directly held-out CE 3.997 transformer 4.429 held-out PPL 54.4 transformer 83.8 generation ~12× 226 ms vs 2,735 ms · 200 tokens ✓ grad check max |∂batched − ∂scalar| = 2.98e−8 gate 3e−8
01 · Overview
PSSA is built around a persistent state, an episodic memory with a hard size limit, and weights that absorb experience during training. Each mechanism is deliberately small enough to measure, ablate, and inspect.
It is a different accounting of context: carry a state, look things up, and consolidate what matters.
A selective diagonal state-space recurrence carries 256 × 16 values left to right. Every token updates that state once; no context window is re-read and no attention matrix is built.
ht = fθ(ht−1, xt)
512 episodic slots hold keys in a Poincaré ball and values in Euclidean space. Reads are an exact softmax over every key; writes rotate round-robin, so the bank can never grow.
m = Σs wsvs
A rank-16 adapter shapes every block output. At consolidation boundaries, a ridge solve folds accumulated statistics into a base matrix — a weight update with no second backward pass.
Abase ← Abase + ridge(H, ΔH)
Scope. PSSA is a 1.54M-parameter research prototype, trained once on a small corpus and measured against a size-matched baseline. Read the numbers as an existence proof, not a production claim. The method and the limits are part of the result.
02 · Architecture
Everything is computed once per token, left to right. No context window is re-read and no attention matrix is materialized. Step through the block below, or turn on narration and listen.
The equations are the actual operations, simplified only where the deep dive goes further: the memory read is a full scan by default, and consolidation runs at explicit boundaries.
Interactive walkthrough
Select a phase, hover a node, or use the narration control.
Phase 01 · project
The token ID indexes a 2,048-entry embedding table. The resulting 256-dimensional vector is the only thing the recurrent core sees; there is no context window to re-read and no positional attention to compute.
embedding
2,048-entry vocabulary · 256 latent dimensions per token
Phase 02 · selective recurrence
A diagonal selective state-space layer carries the context. Its step size, input projection and output projection are read from the current token, so each channel can open slowly or close quickly from one step to the next. The update is elementwise, and total cost stays linear in sequence length.
selective SSM
256 × 16 state persists between tokens · 16 log-spaced time constants, 1.5–200 tokens
Phase 03 · episodic read
The query mixes the current token with the recurrent output. It is projected inside a Poincaré ball, where distance grows toward the boundary and can encode hierarchy. The read scans every one of the 512 keys, scores them by hyperbolic distance, and mixes their Euclidean values.
full-scan read
512 hyperbolic distances per read · optional top-k still scans all keys, then mixes only K values
Phase 04 · adaptation
A sigmoid gate decides how much of the memory read enters the block. A rank-16 adapter then makes a learned correction. At consolidation boundaries, accumulated statistics are solved rather than gradient-stepped: the base weights absorb a closed-form delta.
gate and adapter
rank-16 adapter · ridge consolidation folds accumulated input/output statistics into the base matrix
Phase 05 · prediction & write
The adapted block output is projected to the 2,048-token vocabulary. After the step, the state summary becomes a candidate key and value. A round-robin target slot accepts the write only when the entry is novel enough and the slot has cleared its refractory window.
vocabulary projection
z is the adapted block output · writes require novelty above a threshold and a target slot outside its refractory window
Interactive schematic
Type a few words. The page splits them into visible token chips, animates them through a schematic of the block, and shows a small slice of the state, the memory gate, and the slots that fire. Everything is deterministic and local; nothing is sent to the model. The four stages are a compact summary of the five walkthrough phases: the gate-and-adapt step is folded into Predict.
A schematic forward pass
Token by token · 8 state channels shown · 512-slot memory
embedding · 256 dims
selective update
full 512-key scan
logits + write gate
How much of the memory read enters the block at this step.
03 · Evidence
One seed. 12.7M cleaned WikiText-103 tokens. A 1.54M-parameter PSSA and a parameter-matched transformer trained under the same 30,000-update cosine horizon, then evaluated on 198,939 held-out tokens neither model saw.
The comparison is deliberately narrow. Every number below is reproducible from the repository; the method and its limits are written down in the technical deep dive.
Held-out perplexity
54.4PPL
Against 83.8 for the parameter-matched transformer.
Held-out cross-entropy gap
0.43nats
3.997 for PSSA, 4.429 for the baseline.
CPU training throughput
4.13×
1,716 vs 415 tokens per second on matched work.
Generation speed
12.1× faster
226 ms vs 2,735 ms for 200 generated tokens.
Cross-entropy on supervised target tokens across 64 links. The series is embedded from the training log; hover, tap, or focus the plot to inspect a point.
checkpoint · ≈199k training tokens each
The run ends slightly above its checkpoint-56 low (3.887 → 3.982): per-link variance on a single seed, not divergence. The full 64-point series ships in loss.csv.
| Checkpoint | Training loss | Approx. tokens seen |
|---|---|---|
| 1 | 5.733 | 0.2M |
| 8 | 4.447 | 1.6M |
| 16 | 4.187 | 3.2M |
| 24 | 4.088 | 4.8M |
| 32 | 3.988 | 6.4M |
| 40 | 3.973 | 8.0M |
| 48 | 3.937 | 9.5M |
| 56 | 3.887 | 11.1M |
| 64 | 3.982 | 12.7M |
PSSA against a parameter-matched transformer. Bars are drawn from the measured values and normalized per row; the numbers are the result, not the bars.
| Measure | PSSA | Transformer | Gap | Relative scale |
|---|---|---|---|---|
| Training cross-entropyfinal checkpoint · nats / token · lower is better | 3.98 | 4.43 | −0.45 | |
| Held-out cross-entropy198,939 unseen tokens · nats / token | 3.997 | 4.429 | −0.43 | |
| Held-out perplexityper-token perplexity · lower is better | 54.4 | 83.8 | −29.4 | |
| Next-token accuracytop-1 on the held-out slice · higher is better | 24.1% | 18.0% | +6.1 pts | |
| CPU training throughput2 vCPU · matched fixed work · higher is better | 1,716tok/s | 415tok/s | 4.13× | |
| Generation latency200 generated tokens · lower is better | 226ms | 2,735ms | 12.1× faster |
The limitations are part of the result, not a footnote. Read them before generalizing from any number on this page.
04 · Under the hood
Every figure here is taken from the repository configuration and logs. Nothing is rounded up to sound bigger.
Attention costs grow with context length. PSSA carries a fixed-size state, reads a bounded memory with an exact full scan, and writes with a round-robin slot policy, so per-token cost stays flat no matter how far the context reaches back.
No. It is a different accounting of context, tested at prototype scale against a parameter-matched baseline. The comparison is deliberately narrow and the limits are written down.
At 1.5M parameters, text quality is poor for both PSSA and the baseline. One seed, one corpus, no memory-bank ablation yet. Those are the honest edges of the result.
From the repository: the training logs, the 64-checkpoint loss series in loss.csv, and the evaluation scripts that produced the held-out figures. Re-run them and you get the same table.
05 · Get it
The repository is the artifact: model code, data preparation, loss logs, evaluation scripts, and release notes. The weights are public. If a number here does not reproduce, open an issue.
Model definition, data preparation, loss logs, evaluation scripts, and the gradient-verification harness.
Released checkpoints, configuration, and the evaluation numbers behind this page.
Questions, reproductions, and counterexamples are welcome.
