One token per pass

A language model writes by repetition. One pass through its 24 layers scores every possible next token (the subject of Warp, Shift, Fold); the chosen token is appended to the text, and the pass runs again. Because each pass takes the previous output as input, the passes form a strict sequence: no amount of parallel hardware can begin the eighth token before the seventh exists. It is this dependence, more than the arithmetic, that sets the pace of generation.

The method

Suppose some inexpensive process proposes the next few tokens. A single pass of the full model over the text and the proposals together reveals, at every position, the token the model itself would have chosen there. The proposals are accepted up to the first disagreement, and at that point the model's own token is taken instead. If the first three proposals are correct, one pass has yielded four tokens rather than one.

Why the output cannot change

A proposal is accepted only when the model would have produced exactly that token unaided. The output is therefore identical, token for token, to ordinary generation. The quality of the proposals governs how quickly the text is produced, never what the text is.

Training or answering?

Answering (inference) only. Speculative decoding retrains nothing and leaves every parameter of the model untouched; it is a different way of running a finished model. For that reason it can be applied to models already in service, which largely explains its rapid adoption.

Where the idea comes from

Two groups published the method independently in 2022–23: Leviathan, Kalman and Matias at Google, and Chen and colleagues at DeepMind. Both build on the blockwise parallel decoding of Stern, Shazeer and Uszkoreit (2018), which first used one pass to verify several predicted tokens.

The claim under test

One pass of the model can verify several proposed tokens at once, and the resulting text is identical to ordinary generation, produced in fewer passes. The steps that follow test both halves of that claim on the live model, with the passes counted.

The route

2. One at a time: ordinary generation, with the passes counted.
3. All at once: one pass produces a prediction at every position.
4. You guess: you supply the proposals, and one pass accepts or rejects them.
5. Copy and check: an automatic proposer that copies from the text, run to completion.
6. The real thing: the achievable speed-up, and how the method stays exact when the model samples at random.
7. Name it: the standard terminology and the speed-up formula.
8. Check yourself: four short questions.

How to use this station

Steps 2 to 5 send the text you type to a live model on our server: Qwen2.5-0.5B, the same model used in Warp, Shift, Fold. Each step reports the passes it requested. The server allows 30 passes per minute per visitor, which is ample at a reading pace, and repeating an earlier run costs nothing. Step 6 runs entirely in the browser. The strip above records which steps you have visited.

Ordinary generation advances one token per pass. The panel below makes the loop visible: each request to the model is one full pass over the text so far and returns exactly one new token, which the page appends before making the next request. The counters record passes and tokens as they accumulate.

What the counters show

Passes and tokens rise in lockstep. However fast a single pass may be, the eighth token cannot be started until the seventh is complete, because the seventh is part of its input. A process in which each output is fed back as the next input is called autoregressive.

Where the time goes

On current hardware a pass is limited less by computation than by memory traffic. Every one of the model's parameters, about half a billion here and hundreds of billions in the largest models, must be moved from memory to the processor to produce a single token, and the arithmetic performed on each is trivial by comparison. The processor therefore spends most of a pass waiting for data. Speculative decoding exploits exactly this imbalance, as the next step shows.

Training or answering? Answering. Training never generates text this way; it scores every position of a known text at once, as step 3 explains.

A pass produces more than a prediction for the end of the text. The model carries every token through all 24 layers in parallel, and the same final scoring can be applied at every position. One pass over a sentence therefore yields a prediction of what follows each of its tokens. Ordinary generation keeps only the last of these and discards the rest; the panel below displays them all.

Why every prediction is faithful

Consider the prediction under " is" in "The capital of France is". It is computed from "The capital of France is" alone, even though " Paris" and everything after it passed through the model in the same pass. Within the model, each position may draw on itself and the positions before it, never on those after. This constraint is the causal mask, and it is what makes the method sound: the prediction at every position is exactly what the model would have produced had the text ended there.

Why it costs about as much as one token

As step 2 established, the dominant cost is fetching the parameters. A pass over thirty tokens fetches them once and uses each thirty times. In our measurements on this server, a pass over 5 tokens took about 0.19 seconds and a pass over 30 about 0.30: six times the tokens for less than twice the time. On the GPUs that serve large models the difference is typically greater still, because they have far more arithmetic capacity in reserve.

Training or answering? Both, and the connection is instructive. Training uses precisely this view of a pass: one pass over a training text yields a prediction at every position, each is compared with the token that actually follows, and all of those errors are learned from at once. The technique is called teacher forcing. Ordinary answering uses only the final prediction. Speculative decoding carries the training view into answering by putting the other predictions to use.

The proposals can now be supplied deliberately. Here you act as the inexpensive proposer, called the draft. Type the continuation you expect. The page sends the text and your guess to the model as a single pass, then compares each token of your guess with the model's own prediction at that position. Tokens are accepted while the two agree. At the first disagreement the remainder of your guess is discarded and the model's token is taken instead.

What the result shows

No pass is wasted. Even when the first proposed token is wrong, the pass still supplies the model's own token at that position, so the method never does worse than ordinary generation's one token per pass. Each correct token saves a pass. When the entire guess is correct, the model's prediction after its last token is obtained as well, at no extra cost.

Verification stops at the first disagreement because everything after it was conditioned on a token the model would not have written. The model's predictions at those later positions describe a text that will never exist, and so carry no information.

It is worth trying three cases: a guess that fails early, one that is correct throughout, and one with the wrong spacing (for example "Berlin" with no leading space). Tokens that begin a new word usually include the preceding space, so the last case fails at its first token.

Training or answering? Answering. The model learns nothing from a rejected guess; its parameters are unchanged.

A human draft is far too slow to be useful, so this step uses an automatic one that costs almost nothing. It takes the last two tokens written, finds the most recent earlier place in the text where the same two tokens occur, and proposes the six tokens that followed them there. If no pair matches it tries a single token, and if nothing matches it proposes nothing, so that pass is an ordinary one. The method is called prompt lookup. It is most effective when the answer repeats material from the question, as in rewriting, editing, code changes and summaries that quote. Below, it runs to completion, and every round is one pass of the live model.

What the runs show

On Capitals the copier fails precisely where the answer contributes something new, the country and the city, and succeeds on the repeated scaffolding, " The capital of". It roughly halves the number of passes. On Past tense it copies " the mat and" directly from the original sentence, and the model alters only what the rewrite requires, such as "watches" to "watched". On Spelling it never succeeds: the letters of "orange" appear nowhere earlier in the text, so every pass yields a single token, exactly as in ordinary generation. Apart from the negligible cost of the search, the method is never slower.

The comparison line tests the claim made in step 1: the text produced by guess and check is identical to the text produced one token at a time.

Training or answering? Answering throughout. The copier involves no training at all, only a few lines of string matching, and the model is unchanged. The only quantity that differs is the number of passes needed to reach the same answer.

In deployed systems the draft is usually a small model: a smaller language model from the same family, sharing the same vocabulary of tokens and many times cheaper per pass. It generates a few tokens in the ordinary way, one per pass but quickly, and the large model verifies them all in a single pass. The model on this site, Qwen2.5-0.5B, is commonly used as exactly such a draft for its much larger relatives. Whether the arrangement pays depends on two quantities: how often the draft is right, and what it costs.

Reading the chart

A longer draft can yield more tokens per pass, but only if every token in it is correct, and the probability of that falls with each token added. Each drafted token also carries a small cost. There is therefore an optimal length, typically a handful of tokens. Drafts that are right 70–80% of the time at a few percent of the large model's cost give the two- to threefold speed-ups reported in the literature. As the chance of a correct token falls, the bars sink toward the dashed line, and below it whenever drafting is not free.

When the model samples at random

Until now the model has always taken its most probable token, which is called greedy decoding, and verification has meant asking whether the tokens are the same. Chatbots usually sample instead: they choose the next token at random, weighted by the model's probabilities, which is why the same question can receive different answers. The question "would the model have written this?" then has no yes-or-no answer. A short rule restores exactness:

• The draft proposed a token to which it gave probability q; the large model gives it probability p.
• If p ≥ q, accept it. Otherwise accept it with probability p / q.
• On rejection, draw a replacement from the remainder: the tokens the large model favours more than the draft did, in proportion to the difference.

The rule appears too simple to be exact. The panel below tests it by drafting and verifying ten thousand tokens and comparing how often each word emerges with the large model's own probabilities.

Why it is exact

Consider a single word. It can emerge in two ways: drafted and accepted, or drawn from the remainder after a rejection. The first route contributes the smaller of p and q. The remainder is exactly the part of p that the first route falls short of, and it is drawn in proportion to that shortfall. The two routes sum to p for every word. A better draft does not change the result; it only reduces how often the remainder is needed.

Other kinds of draft

• Prompt lookup, as in step 5: essentially free, and effective when the output repeats the input.
• A small model of the same family: the arrangement studied in the two original papers (Leviathan, Kalman and Matias at Google; Chen and colleagues at DeepMind).
• Additional heads on the large model itself, trained to predict two, three or four tokens ahead from its final layer; Medusa (Cai and colleagues, 2024) and EAGLE (Li and colleagues, 2024) are well-known examples. Training those heads is training, but the large model's own parameters stay frozen.

Training or answering? Answering. Only the additional-heads variant involves training, and that training touches the heads alone.

Every idea on this page has already appeared in the steps above. The table gives the standard names, which open the research literature, and the symbols, which are shorthand for the same operations.

The speed-up formula

With a draft of $\gamma$ tokens, each correct with probability $\alpha$, one pass of the large model produces on average $$\frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$$ tokens, never fewer than one. If one draft token costs $c$ passes of the large model, the speed-up is that number divided by $\gamma c + 1$. This is the quantity plotted in step 6 (Leviathan, Kalman and Matias, 2023).

What this does not yet explain

• Our passes reread the whole text each time. Production servers keep a KV cache, a store of the work already done for earlier tokens, so an ordinary pass processes only the one new token and a verification pass only the proposed ones. The pass counts are the same either way; the cache will have a station of its own.
• How does a position "look back"? Through attention, the part of each layer that draws on earlier tokens; the causal mask is a constraint within it. Attention is the subject of the next station.
• Why do large models wait on memory? The balance between fetching parameters and performing arithmetic shapes much of how models are served, and it is treated more fully alongside batching.

Four questions on the station's central ideas. Each should be answerable without referring back.