How LLMs Work

Step 1: Tokenization

Before a language model can understand your text, it needs to break it into small pieces called tokens. But here's the thing — the model doesn't split on words the way you'd expect. It uses an algorithm called byte-pair encoding (BPE) that finds the most efficient way to represent text using a fixed vocabulary of about 50,000 subword pieces.

Common words like "the" and "is" get their own token. But rarer words get split into pieces: "unforgettable" might become ["un", "forget", "table"]. This is why LLMs can handle any word — even made-up ones — without needing an infinitely large vocabulary. Every token maps to a unique integer ID (like "the" → 464 in GPT-2).

BPE works by starting with individual characters and repeatedly merging the most common adjacent pairs. After thousands of merges, you get a vocabulary that efficiently balances between character-level flexibility and word-level efficiency. English text averages about 4 characters per token.

This has real consequences. Tokenization is why models struggle to count letters in words (they see "strawberry" as ["straw", "berry"], not individual characters), why some languages cost more tokens than others, and why code sometimes tokenizes strangely. Try it below — switch between Simple Split and BPE to see the difference:

Step 2: Embeddings

Each token gets converted into an embedding — a dense vector of hundreds or thousands of numbers that encodes the token's meaning. GPT-3's embeddings are 12,288 dimensions long. These aren't hand-crafted; the model learns them during training by adjusting the numbers to minimize prediction errors.

Here's the fascinating part: words with similar meanings end up near each other in this high-dimensional space. "Cat" and "dog" are close together. "King" and "queen" are close. But "cat" and "democracy" are far apart. The space organizes itself into semantic neighborhoods — without anyone telling it to.

Even more remarkably, the directions in embedding space encode relationships. The vector from "man" to "king" points in roughly the same direction as the vector from "woman" to "queen". This means you can do vector arithmetic: king − man + woman ≈ queen. The model has discovered gender, royalty, tense, and hundreds of other concepts — just from predicting text.

Below is a 2D projection of a real embedding space. Explore the clusters, try vector arithmetic, and check how similar different word pairs are:

Step 3: Attention

This is where the magic happens. The attention mechanism is the core innovation of the transformer architecture — and arguably the most important idea in modern AI. It lets each token look at every other token in the sequence and decide which ones are relevant for understanding it.

Here's how it works mechanically: for each token, the model computes three vectors called Query, Key, and Value. The query asks "what am I looking for?", the key says "here's what I contain", and the value says "here's the information to pass along." Attention weights are computed by comparing each query against all keys, then used to create a weighted combination of values.

But there's a twist: modern models don't use just one attention pattern. They use multi-head attention — typically 32 to 128 heads running in parallel, each learning to detect different types of relationships. One head might track syntax (linking subjects to verbs). Another resolves pronouns ("she" → "Alice"). Another handles positional patterns. Together, they build a rich understanding of the input.

This is what allows the model to understand that "bank" means different things in "river bank" vs "bank approved the loan." Try the different sentences below — click a word to see its attention pattern, and toggle between heads to see different perspectives:

Step 4: The Transformer Stack

A single round of attention isn't enough. Modern LLMs stack dozens to over a hundred transformer blocks, each one refining the model's understanding. GPT-3 has 96 layers. GPT-4 is estimated to have even more. Each layer consists of the same four operations: multi-head attention, add & normalize, feed-forward network, add & normalize.

What makes this work is the residual connection — a seemingly simple idea with profound consequences. Instead of replacing the input with the attention output, the model adds the attention output to the original input. This creates a "skip highway" that lets information flow through all 96 layers without degrading. Without residuals, training deep networks is nearly impossible.

The feed-forward network in each block is where most of the model's "knowledge" lives. It's a simple two-layer neural network, but when you multiply it by 96 layers with billions of parameters, it stores an enormous amount of factual associations, linguistic patterns, and reasoning heuristics. Researchers have found that specific neurons activate for specific concepts.

As information flows through the layers, it transforms from shallow surface patterns (punctuation, word shapes) to deep semantic understanding (reasoning, world knowledge). Explore the pipeline below, toggle residual connections on and off, and use the depth slider to see what the model has learned at each level:

Step 5: Next-Token Prediction

After passing through all those transformer layers, the model's final job is deceptively simple: predict the next token. The last layer's output — a vector of 12,288 numbers — gets projected to vocabulary size (50,257 for GPT-2) through a linear transformation, producing a raw score (logit) for every possible next token.

These logits are then passed through a softmax function that converts them into a probability distribution — every token gets a probability between 0 and 1, and they all sum to 1. For factual prompts like "The capital of France is", the distribution is extremely sharp — "Paris" gets 92% of the probability mass. For open-ended prompts like "The meaning of life is", it's much flatter — many tokens are plausible.

This shape — sharp vs flat — is quantified by a measure called entropy. Low entropy means the model is confident. High entropy means many tokens are equally plausible and the model is "unsure." Notice how different prompt types produce radically different distributions:

Try the different prompts below. Pay attention to the long tail — even the top 10 predictions rarely account for 100% of the probability. The other 50,000+ tokens each get a tiny sliver:

Step 6: Temperature & Sampling

You've seen how the model produces a probability distribution over next tokens. But how does it choose which one to use? This is where temperature comes in — the one parameter that you, the user, actually get to control.

Temperature works by scaling the logits (raw scores) before the softmax function. Mathematically, it's simple: divide every logit by the temperature value T. Low T (like 0.2) makes the differences between logits larger, sharpening the distribution so the top token dominates. High T (like 1.5) compresses the differences, flattening the distribution so more tokens have a chance of being selected.

At temperature 0 (or very close to it), the model always picks the single most probable token — this is called greedy decoding. The output is completely deterministic: same input, same output, every time. At temperature 1.0, the probabilities are used as-is. Above 1.0, you start getting increasingly surprising and creative — but potentially incoherent — output.

In practice, most APIs also offer top-k and top-p (nucleus) sampling to further shape the distribution. Top-k only considers the k most likely tokens. Top-p keeps the smallest set of tokens whose cumulative probability exceeds a threshold. Both prevent the model from picking extremely unlikely tokens. Drag the slider below to see how temperature reshapes the distribution in real time:

Step 7: The Generation Loop

Here's how it all comes together. To generate text, the model takes your prompt, runs the entire pipeline (tokenize → embed → attend → transform → predict → sample), selects one token, appends it to the sequence, and repeats the entire process from scratch. Each new token is fed back in as input. This is called autoregressive generation — the output feeds back into the input.

This means that for a 100-token response, the model runs the full inference pipeline 100 times. Each pass is independent — the model has no "memory" of previous computations beyond what's in the token sequence itself. Every new token requires re-processing the entire sequence from the beginning (though in practice, KV caching avoids redundant computation for earlier tokens).

At each step, there's not just one possible next token — there are thousands of plausible continuations. The model picks one based on the probability distribution and temperature settings. A different random seed or slightly different temperature would produce a completely different text. The "ghost" alternatives you'll see below represent paths not taken — parallel universes of text that could have been.

Generation continues until the model produces a special end-of-sequence (EOS) token, or until it hits a maximum length. Watch the process step by step below — and try clicking alternative tokens to explore how a single different choice cascades into a completely different continuation:

How It Learned: Training

Everything you've seen so far is inference — what happens when you use a trained model. But how did the model learn all this? Training is where billions of parameters get their values, and it happens in two major phases.

Phase 1: Pre-training. The model is shown hundreds of billions of tokens of text — web pages, books, code, Wikipedia, academic papers. For each token in the training data, the model tries to predict what comes next. When it's wrong, the error signal flows backward through the network via backpropagation, nudging all 175 billion parameters slightly in the direction that would have produced a better prediction. Repeat this trillions of times.

Phase 2: Fine-tuning (RLHF). The pre-trained model can continue text, but it doesn't follow instructions well. Fine-tuning with Reinforcement Learning from Human Feedback transforms it into a helpful assistant. Human evaluators rank model outputs, a reward model learns their preferences, and the LLM is trained to maximize that reward. This is what makes it answer questions instead of just continuing text.

The loss curve below shows how prediction error decreases over training. Early on, the model is nearly random. As training progresses, it first learns common words, then grammar, then facts, then reasoning. The comparison panel shows the dramatic difference between a pre-trained base model and one that's been fine-tuned:

What It Can't Do

Understanding how LLMs work also means understanding their fundamental limitations. These aren't bugs to be fixed — they're inherent to how the technology works. A next-token predictor trained on text patterns will always have certain blind spots.

Hallucinations. LLMs generate text that is statistically plausible, not factually verified. If a confident-sounding answer fits the pattern, the model will produce it — even when it's completely fabricated. It might invent people, cite papers that don't exist, or state false facts with absolute confidence. The model has no internal mechanism to distinguish what it "knows" from what it's making up.

Limited context. Every model has a finite context window — a maximum number of tokens it can process at once. Anything beyond that window is completely invisible. In a long conversation, the model literally cannot see what was said earlier. Newer models have expanded this dramatically (up to 200K tokens), but it's still a hard limit.

Reasoning failures. LLMs are surprisingly bad at certain tasks that seem easy to humans. Counting letters in words, basic arithmetic, logic puzzles — these often trip them up because the model is pattern-matching, not actually computing. The classic bat-and-ball problem fools most LLMs because the wrong answer appears far more often in training data than the right one. Explore these limitations interactively below:

The Big Picture

Let's zoom out. Every time ChatGPT writes a sentence, Claude answers a question, or Copilot suggests code, this entire pipeline runs — for every single token. Text comes in, gets tokenized, embedded, processed through attention layers, and a single next token comes out. Then it loops. A 500-word response requires roughly 375 complete passes through the entire network.

The "intelligence" isn't in any one step — it's in the billions of trained parameters that shape how attention flows, how representations transform layer by layer, and which tokens get predicted. An LLM is, at its core, the world's most sophisticated autocomplete. But "just autocomplete" undersells it — the depth of pattern recognition across 96 layers, trained on most of the internet, produces emergent capabilities that continue to surprise researchers.

What's remarkable is that the entire architecture is built from simple, differentiable operations — matrix multiplications, additions, softmax. There's no special "understanding" module, no knowledge database, no explicit reasoning engine. Just prediction pressure, scale, and data. Whether that constitutes genuine understanding or merely the most convincing imitation of it — that's a question for philosophers.

Click any stage in the pipeline below to jump back to that section. And remember: every token you've read in this explainer could have been generated by the very process it describes.

Frequently Asked Questions