Independent research project building a non-Transformer language model: gated-delta-rule blocks, a content-blind "nucleus" channel, a memory pool that keeps spawning, merging and reorganizing itself during training and at inference, live self-monitoring signals, and an ESN residual path.
No fluff tiers, no hype benchmarks โ honest results, including where a plain Transformer still wins.
The project doesn't lead with a PPL win over Transformers โ it isn't one, at these scales. What it reports, including the parts that did not turn out as hoped: the memory is load-bearing but its benefit is structural, not content-specific, and perplexity keeps falling as the context grows well past the trained length, where a plain Transformer's rises.
Full findings, methodology and numbers: see each model's card below, or sencelium.com/results ยท benchmarks/results.md.
Non-standard architecture โ not AutoModel-compatible. Each model card
has exact instructions for loading via the training/eval code in the
GitHub repo.