Attention is the core mechanism of transformers (BERT, GPT, Llama…). Each word (token) is not processed in isolation: it looks at the others and decides how much each one matters when building its own meaning. So in “The cat… because it was tired”, “was” can learn to focus on the cat. Here you see it two ways: the token-to-token matrix (each row sums to 1 = 100%) and the arcs over the sentence. Type your own sentence and experiment.
To decide who to look at, each token emits three vectors: Query (Q, “what I'm looking for”), Key (K, “what I offer”) and Value (V, “what I contribute if I'm attended to”).
• The weight of token i on j is the dot product of i's Query with j's Key.
• It is divided by √dk (scaling, so the numbers don't blow up) and passed through softmax, which turns each row's weights into percentages that sum to 1.
• In short: A = softmax(Q·Kᵀ / √dk). This tool draws that matrix A; a real model would use those weights to mix the Value vectors.
A transformer doesn't have ONE attention, but several heads (here 4) in parallel. Each head uses different Q/K projections, so it learns to focus on different relationships: one on the previous word, another on the subject, another on agreement… and then they are combined. Switch Head with buttons 1–4 and watch how the pattern changes completely for the same sentence.
Models that generate text (GPT) predict the next word, so they can't look at the future: each token only attends to itself and the previous ones. Turn on Causal mask (GPT) and you'll see the whole upper-right half of the matrix go dark — those future positions are set to 0 before the softmax. Without the mask, as in BERT, each token sees the whole sentence, to its left and right.
The attention matrix is a token × token heatmap: the row is the token doing the looking (from) and the column the token being looked at (to).
• The more intense the blue of a cell, the higher the weight; white is almost zero.
• Each row sums to 100% (it is a distribution).
• Hover a cell to read the exact value (e.g. cat → cat: 40%).
• The diagonal usually lights up: a token almost always attends fairly strongly to itself.
Over the sentence strip, click a word and arcs appear toward the words it looks at, taken from that row of the matrix. The thickness and opacity of each arc are proportional to the weight (weights below 2% are hidden). Click again to deselect. It's the same information as a heatmap row, but drawn over the text so you can read it at a glance.
1. Type a sentence in the field above (it's tokenized by words and punctuation, up to 40 tokens).
2. Pick a Head (1–4) to see different patterns.
3. Toggle the Causal mask to compare GPT vs. BERT.
4. Click a word (on the strip or on its row label) to draw its arcs.
5. Hover the heatmap to read exact weights. Everything recomputes instantly, in your browser.
This visualizer is educational and deterministic (offline): each token's vectors are generated by hash (equal tokens → same vector) plus a sinusoidal positional encoding, and the Q/K projections are fixed with a seed, so the result is reproducible. The attention computation (scaled dot product + softmax, with optional causal mask) is the real one. A true model learns Q/K/V from data and stacks dozens of layers and heads. Related: Tokenizer, Mini-LLM and Dimensionality reducer.
Click a word to see where it looks:
This is the attention mechanism (scaled dot product + softmax). A real model learns the Q/K/V projections and stacks dozens of layers and heads, but the idea is this. Related: Tokenizer, Mini-LLM and Dimensionality reducer.