Running a Language Model on a 1975 6502

A developer trained a tiny Mamba-based autoregressive language model and wrote an inference engine for the MOS 6502, the 8-bit processor from 1975 that powered the BBC Micro and Apple II. The model runs on a BBC Model B with 32KB RAM, generating text like: "once upon a time tom and lily saw things lily were sad her house he heartd them ilily and tom said yes she saw a little girl smiled tom was so excited her mom said yes".

The Constraints

The 6502 is an 8-bit CPU with no native multiply instruction. The entire model and inference code must fit in 25KB of user-space memory. The final split: 9KB of inference code and 13KB of model weights. The developer used CC65 to compile C to 6502 assembly, trained the model on a MacBook, and loaded the binary via a custom 3.5mm-to-tape cable that convinces the BBC Micro it's reading a tape drive.

Why BitNet?

BitNet quantizes weights to ternary values (-1, 0, +1), reducing matrix multiplication to adds and subtracts. On the 6502, a multiply-accumulate costs ~150 clock cycles, but a ternary accumulate takes ~30 cycles. Each ternary parameter needs only 1.58 bits, so 4 parameters can be packed per byte. With 13KB, that's 52,000 parameters.

The developer chose 4 parameters per byte over 5 because unpacking 5 requires repeated floor-divide-by-3 (not a native 6502 instruction), while 2-bit chunks only need a right shift.

Why Not Attention?

Transformers use attention, which requires a KV cache that grows with context. On a 32KB RAM budget, that would eat memory needed for weights. Recurrent models like Mamba keep a fixed-size state, so memory usage stays constant per token. The developer chose Mamba over GRUs because GRUs suffer from vanishing/exploding gradients, especially with ternary weights. Mamba's per-channel decay is computed at inference time and never exceeds 1, preventing explosion.

Training with Ternary Weights

The model is trained in full float32, but quantized to ternary during the forward pass. Gradients flow in full precision using a straight-through estimator:

def ternary_quantize(w: torch.Tensor) -> torch.Tensor:
    q = torch.clamp(torch.round(w), -1.0, 1.0)
    return w + (q - w).detach()

The final LM head is kept in int4 for better output resolution.

16-Bit Accumulation and Learned Scaling

Activations are stored as 8-bit. Each accumulator term is ≤128, so up to 256 terms can be summed into 16-bit without overflow. After each layer, activations are clipped back to 8-bit. A simple clip loses dynamic range, so the developer uses a learned shift:

def activation(x: int16, shr: int) -> int8:
    return clip(x >> shr, min=-128, max=127)

The scale parameter shr is allowed to vary in the first half of training, then frozen.

The Ternary Matmul Loop

The core inference primitive is a ternary-by-char matrix multiplication. Here's the annotated C code:

void ternary_linear(struct ternary_matrix *W,
                    struct char_matrix    *x,
                    struct int_matrix     *bias,
                    unsigned char          shift,
                    struct char_matrix    *out) {
    unsigned char w_packed = (W->width + 3) >> 2;
    for (i = 0; i < W->height; i++) {
        for (j = 0; j < x->width; j++) {
            a = bias->data[i];
            for (k = 0; k < w_packed; k++) {
                b = W->data[i * w_packed + k];
                for (l = 0; l < 4; l++) {
                    switch (b & 0b11) {
                        case 0b00: break;
                        case 0b01: a += x->data[j + x->width * (4 * k + l)]; break;
                        case 0b10: a -= x->data[j + x->width * (4 * k + l)]; break;
                    }
                    b = b >> 2;
                }
            }
            out->data[i * out->width + j] = shift_sat_int8(a, shift);
        }
    }
}

Sampling Without Exponential

The 6502 can't compute exp(), so the developer uses a lookup table for softmax sampling. With temperature T=0.9, the lookup is [255, 83, 27, 9, 3, 1, 0, 0, ...]. After subtracting the max logit, they sample using a pseudo-random 16-bit integer.

Running It Yourself

The developer provides a link that boots a BBC Micro in your browser, loads the UEF tape image from GitHub, and auto-types the commands. Generation takes a few minutes.

Verification

The sim65 emulator provides a parity check between the C inference binary and the Python reference implementation. The full engine is also tested on the jsbeeb emulator before running on real hardware.

What This Means for You

This project demonstrates that modern ML techniques can be adapted to extreme hardware constraints. The techniques here — ternary quantization, recurrent architectures, learned scaling — are directly applicable to edge devices with limited memory and compute. If you're working on embedded ML or just want to see how far you can push a language model, this is a masterclass in optimization.

Key Takeaways

Try running the model in your browser — it's a fun way to appreciate how far we've come.