glasswing
A small language model you can watch think. It learned to write children's stories, and it runs right here in your browser on an inference engine written from scratch in Rust, with no ML framework. While it writes, every layer, neuron and attention head is visible.
- parameters
- 15.7M
- validation loss
- 1.18
- trained on, one GPU
- 1.3B tokens
- download, f16
- 31.5 MB
Watch it write
Each token climbs the tower from the embedding at the bottom to the prediction at the top. The grids are the MLP neurons of the eight layers, the orbs beside them the attention heads. Hover any word in the story to see what the model was predicting there and which earlier words it looked at.
The model downloads once (31.5 MB) and then runs entirely in your browser. Nothing you type leaves your machine.
- ●
- MLP neuron, positive
- ●
- MLP neuron, negative
- ◉
- attention head: brighter = more focused
- ┃
- residual stream (384 numbers per layer)
How it works
A Llama-style transformer (RMSNorm, rotary positions, SwiGLU), 15.7M parameters, trained with PyTorch on one RTX 3060 Ti. The browser runs a separate implementation: the same model written out in plain Rust (its own safetensors reader, matrix maths, attention loop and KV cache) and compiled to WebAssembly with SIMD. A test in CI compares every intermediate tensor against PyTorch, so the two can't drift apart.
The circuit inside
The model learned an induction circuit: two previous-token heads in layer 3 (L3H3, L3H1) feed an induction head in layer 4 (L4H4), which looks up what came after the last occurrence of the current word. Switch the three off and copying gets measurably worse while three random heads barely matter; in a name task, activation patching shows the name moving to the last position in layers 5 to 7, the same layer the logit lens says the answer appears. The preset in the lab above switches exactly these heads off. Read the write-up.
The same engine, serving a real model
The engine also loads Hugging Face checkpoints: SmolLM2-135M-Instruct runs on it at glasswing.takodana.dev, served from my k3s cluster through the Cloudflare Tunnel, with greedy answers identical to the reference implementation. It's public without a login, so it limits itself: a token bucket per visitor, two generations at a time, 256 tokens per answer, prompts never logged, time to first token and tokens/s in Prometheus.
What I broke on purpose
Every milestone ends with one deliberate failure, made and fixed through Git, to see what catches it.
- A loss that looked too good. With the causal mask removed, validation loss fell to 0.55 instead of 2.96, and the model wrote “did did but very both both”. It had learned to copy the next token, which doesn't exist yet when you generate.
- A transposed matrix with the right shape. Layer 1's query weights loaded transposed pass every shape check: they're square. The stage-by-stage parity test named the exact tensor that diverged.
- A bug that only changes the story. RoPE positions off by one, only when decoding a single token through the KV cache. The full-sequence test stayed green and generation stayed fluent; it just wrote a different story than PyTorch. A test that splits the sequence caught it.
- My own size estimate. The first plan said 15M parameters for a shape that is 7.2M with our vocabulary. A test on the parameter count caught it before training.
Read the series
Six posts explain glasswing from start to end for readers who have never looked inside a language model, each with interactive figures built from the real model.
- part 1 How a language model reads Bytes, byte-pair encoding replayed on your own sentence, and a map of the model's words.
- part 2 What happens inside Attention, the causal mask, real attention heads, rotary positions, neurons with a job.
- part 3 Teaching it to write The loss, walking downhill, and the training run checkpoint by checkpoint.
- part 4 A model in your browser Rebuilding it in Rust, checking every step against PyTorch, and making it fast.
- part 5 The circuit inside Finding an induction circuit, and proving the model uses it.
- part 6 Serving a model to strangers SmolLM2 on the same engine, rate limits, queues, and a live demo.
Milestones
- M0 · doneRust workspace, CI
- M1 · doneBPE tokenizer, reference model, golden fixtures
- M2 · doneRust forward pass, parity with PyTorch
- M3 · doneTrained on TinyStories, KV cache, sampling
- M4 · doneWebAssembly engine, f16 weights
- M5 · doneThis visualization
- M6 · doneInterpretability: ablation, logit lens, an induction circuit
- M7 · doneSmolLM2-135M served on k3s, public