Lab
The real Rust model, compiled to WebAssembly, running in this tab. Nothing is simulated in JavaScript: every parameter count, loss, gradient and attention map below comes from the same code the tests check.
Loading the engine…
Model calculator
Pick a preset or edit the architecture. The counts come from the Rust code that builds the model, so they are exact.
Train
The task is memorization: random token sequences the model has to learn to predict. A model that cannot do this has a bug; this is the same task the smoke test uses.
Gradient norm by parameter
The task
Attention
Each row shows how much one position reads from every earlier position (a row sums to 1). The empty upper triangle is the causal mask. Train a little first and watch the patterns sharpen.
Selected head
All heads in this layer
Rotary position embeddings
Each pair of channels is rotated by an angle that grows with position, fast for the first pairs and slowly for the last. This is the table the model uses.
cos of the rotation angle (rows are positions, columns are channel pairs)
Four channel pairs across positions
Token embeddings
The embedding table projected onto its two strongest directions. Create a model on the Train tab, then train it and watch the tokens move.