How a Large Language Model Works
An interactive 3D walkthrough of a working GPT-style transformer — watch a real (tiny) model run inference, token by token, layer by layer. Everything is self-hosted and runs locally in your browser: no external downloads, no tracking.
Why learn the internals?
Prompt engineering, fine-tuning, and debugging model behaviour all get dramatically easier once you can picture what happens between typing a prompt and reading a reply. Scroll through the 3D model below to see every matrix multiply, then read the stage-by-stage summary underneath.
The 3D model
Drag to orbit · scroll to zoom · use the sidebar on the right to step through the walkthrough.
This 3D visualization requires WebGL2. Please open this page in a recent version of Chrome, Firefox, Safari, or Edge.
The pipeline, stage by stage
The animation above is a client-side WebGL render, so here is the same journey in words — from raw text to a sampled next token.
1 · Tokenize
Raw text is chopped into tokens — usually subword pieces — and each token is mapped to an integer ID via a fixed vocabulary. From here on the model only ever sees these integers, never the letters.
2 · Embedding
Each token ID looks up a row in a learned embedding table, turning it into a vector of numbers. A positional embedding is added so the model knows each token's place in the sequence.
3 · Layer Norm
Before each sub-layer, every token vector is normalized to mean 0 and unit spread, then rescaled by two learned parameters. This keeps values well-behaved so very deep networks train reliably.
4 · Self-Attention
Each token forms a Query, Key, and Value. The dot product of Queries and Keys decides how much each token attends to the others; a softmax turns those scores into weights used to blend the Values.
5 · Attention Matrix
Laying out every token's attention to every other token gives a triangular matrix (future tokens are masked out). Many heads run in parallel, each learning a different relationship between tokens.
6 · MLP (Feed-Forward)
After attention mixes information across tokens, each token is processed on its own by a two-layer network that expands, applies a non-linearity (GELU), and projects back down. Most parameters live here.
7 · Transformer Block
Layer-norm, attention, residual add, layer-norm, MLP, residual add — one block. A GPT is just this block repeated (12 times in GPT-2 small, 96 in GPT-3), each pass refining the token vectors.
8 · Unembedding & Softmax
The final token vector is multiplied by the unembedding matrix to produce one logit per vocabulary token, then softmax turns those logits into a probability distribution that sums to 1.
9 · Sampling
One token is sampled from the distribution — tuned by temperature, top-k, or top-p — appended to the input, and the whole pipeline runs again. That autoregressive loop is how a model writes text.
Frequently asked questions
What model does this visualization show?
A tiny GPT-style network (nano-GPT) trained to sort the letters A, B, and C — the demo model from Andrej Karpathy's minGPT. It is small enough to render every weight and activation, yet uses the exact same architecture as GPT-2 and GPT-3.
Does anything get uploaded or downloaded from a server?
No. The model weights, font atlas, and all rendering assets are served from this site itself and run entirely in your browser via WebGL. Nothing you do is uploaded, logged, or sent to any third party.
Why can't I see the 3D model?
The visualization needs WebGL2, which every modern browser supports. If the canvas stays blank, try a recent version of Chrome, Firefox, Safari, or Edge, and make sure hardware acceleration is enabled.
How does this relate to real models like ChatGPT?
The pipeline is identical — tokenize, embed, stack transformer blocks, unembed, softmax, sample. Frontier models just scale it up: bigger vocabulary, wider vectors, more heads, and far more layers. GPT-3 has 175 billion parameters running this same machinery.
Credits & license
The 3D visualization is based on the open-source llm-viz project by Brendan Bycroft (MIT license), self-hosted here with no external dependencies. The demo model is the A/B/C sorting nano-GPT from Andrej Karpathy's minGPT.