Autoregressive Language Model on the 6502 Processor

Autoregressive Language Model on the 6502 Processor June 2026 tl;dr - I trained a tiny Mamba-based autoregressive language model and wrote an inference engine to run it on the 8-bit 6502 processor (from 1975, with 32KB RAM). Running it on my dad's BBC Micro generated the text below. once upon a time tom and lily saw things lily were sad her house he heartd them ilily and tom said yes she saw a little girl smiled tom was so excited her mom said yes The MOS 6502 is an 8-bit microprocessor released in 1975, powering the BBC Micro and the Apple II. I am lucky enough to have access to my dad's BBC Model B from the 80s; I wanted to see, using modern machine learning, what the strongest language model we could fit on this machine was. Unsurprisingly, this poses significant challenges. The model weights and inference code need to be contained within 25KB of user-space memory — my final configuration was 9KB inference code and 13KB model weights. The CPU only operates on an 8-bit integer datatype, and doesn't include multiplication in its instruction set. CC65 is used for the inference code, enabling compilation of C to the 6502 instruction set. A binary of a model trained on my MacBook can then be written to the BBC Micro using PlayUEF and a custom 3.5mm-to-tape cable I DIY'ed. This convinces the BBC that it's listening to a tape drive, while my laptop plays audio out of its headphone jack. The sim65 emulator allows a parity check between the C inference binary and the reference Python model implementation. The full inference engine can be tested on the jsbeeb emulator before running on the BBC Micro. You can run it yourself — the link below boots a BBC Micro in your browser, loads the UEF tape image straight from GitHub, and auto-types the commands to run the model. No emulator install required (note that generation takes a few minutes). Run BitNet on a BBC Micro → Modelling The goal of this project is to build an autoregressive language model — a language model that produces tokens one-by-one, similar to frontier language models. The model is a function $f$ that produces the next token from the existing context: $$ f: \text{'the cat sat on the ma'} \mapsto \text{'t'} $$ In large-scale language modeling, a 'token' would be a word or sub-word part. For this post, the vocabulary (list of tokens) used will be 26 letters plus the ' ' character. For models on this small scale, a larger vocabulary (eg. word or subword vocab) would lead to the vocabulary encoder / decoder layers consuming too much of the parameter budget. An embedding layer maps tokens into the hidden dimension (dim=56 in our case) of the model. 3Blue1Brown's video on neural networks is great for understanding how spatial token embeddings work. $$ g: \{\text{a}, \text{b}, ... , \text{z}, '\text{ }'\} \to \mathbb R^{56} $$ Once tokens are mapped to our high dimensional space, mixing layers are used to model recurrent dependencies between tokens (see recurrent layers). BitNet BitNet was introduced as a method for fast inference on CPU. A matrix multiplication $Y = XW$ is a set of dot products of rows of $X$ with columns of $W$: $$Y_{ij} = \sum_k X_{ik} W_{kj}$$ BitNet quantizes $W$ such that its values lie in the ternary set $\{-1, 0, 1\}$. This reduces the dot product to a sequence of add/subtract operations: $$ \begin{align*} Y_{ij} &= X_{i1} W_{1j} + X_{i2} W_{2j} + \cdots + X_{in} W_{nj} \\ &= X_{i1} - X_{i2} + ... - X_{in}\\ \end{align*} $$ The 6502 processor's instruction set doesn't contain multiply: instead, a multiply is built from repeated bit-shift-and-add operations. A single 8×8 multiply and accumulate would cost