HQAI Engineering HQ
Search
Checking sign-in...

Speculative decoding

Concept8 min readadvanced

What you’ll understand

  • Explain how a draft model and target model cooperate during speculative decoding
  • Understand why accepted draft tokens can improve decoding throughput
  • Identify the workload and deployment conditions that determine real speedups

Helpful first: Inference — how models generate output · LLM architecture and attention

Autoregressive language models normally generate one token at a time. Each token requires another model step, so long responses can be slow even when the prompt has already been processed.

Speculative decoding tries to generate several tokens per expensive target-model step. A smaller or cheaper draft model proposes a short run of tokens. The full target model checks those proposals in parallel and accepts as many as its own probability distribution allows.

When many proposed tokens are accepted, the target model advances several positions at once. When a proposal is rejected, generation falls back to the target model at that position.

Draft, verify, accept

The basic process is:

  1. The draft model proposes several future tokens.
  2. The target model scores those positions in one verification pass.
  3. An acceptance rule keeps a valid prefix of the draft.
  4. At the first rejection, a replacement token is sampled from a corrected target distribution.
  5. The process repeats from the new prefix.

The exact acceptance math matters. Correct speculative sampling is designed to preserve the target model’s output distribution. It is not simply “use the draft model when it looks confident.”

Some systems use a small separate model. Others use extra prediction heads, an early-exit path, or a tree of candidate continuations. The shared idea is cheap proposal followed by authoritative verification.

A simple example

Suppose the current text is:

The deployment failed because

The draft model proposes:

the database connection timed out

The target model verifies all five proposed positions. It accepts:

the database connection

At the next position, the target distribution does not accept timed. The algorithm samples a replacement such as was, producing:

The deployment failed because the database connection was

The draft model then proposes again from that updated prefix.

The target model remains authoritative. The draft only suggests work that may be accepted.

One speculative decoding verification step The current prefix, The deployment failed because, enters a small draft model. The draft proposes five tokens: the, database, connection, timed, out. The target model verifies all proposed positions in one pass. The first three tokens are accepted. Timed is rejected at the fourth position, so out is not committed. The target supplies a corrected replacement token, was. The committed text becomes The deployment failed because the database connection was, and the loop repeats from that prefix. Current prefix "...failed because" Draft model cheap proposal Proposed block the | database | connection | timed | out five speculative positions Target model verifies all proposed positions in one pass the database connection timed out accepted prefix first rejection stops commit committed: "...because the database connection was" repeat
The target accepts a valid draft prefix, supplies the first corrected token, and remains authoritative for the committed output.

Why verification can be faster

During normal inference, decode is sequential: one target-model step produces one new token.

Speculative decoding uses extra compute from the draft model but may reduce the number of sequential target-model steps. Modern accelerators can score several proposed positions in one pass more efficiently than running the target model separately for each position.

The speedup depends on:

  • How cheap the draft is compared with the target.
  • How many draft tokens are proposed at once.
  • How often the target accepts them.
  • Batch size and server load.
  • Memory bandwidth and kernel efficiency.
  • The cost of moving cache state and synchronizing models.

If acceptance is low, the system pays for draft work that is frequently discarded.

Acceptance rate

Acceptance rate is the share of proposed tokens that the target accepts. A draft model that predicts similarly to the target tends to achieve a higher rate.

Acceptance can fall when:

  • The prompt asks for a rare domain or language.
  • Sampling temperature is high.
  • The target model is much stronger than the draft.
  • The draft uses a different tokenizer or incompatible vocabulary.
  • The text changes direction frequently, such as creative writing.

A larger draft may improve acceptance but also cost more. The best draft is not necessarily the smallest or most accurate model. It is the model that produces useful accepted tokens for the lowest total serving cost.

Output quality and exactness

There are two different goals:

  1. Distribution-preserving sampling: the algorithm produces samples from the same target distribution, within the assumptions and numerical behavior of the implementation.
  2. Bit-for-bit identical output: the exact same random sequence and floating-point execution produce identical tokens.

The first is the usual theoretical goal. The second is harder in real systems because parallel execution, random-number consumption, kernels, and numeric precision can differ.

For greedy decoding, implementations can often verify draft tokens against the target’s highest-probability token. For sampling, the acceptance and correction rules are more involved.

Do not assume every feature called “speculative decoding” preserves output behavior in the same way. Check the serving engine’s algorithm and evaluation.

KV caches and memory

Both draft and target models need state for the current prefix. In transformer serving, this usually includes a KV cache for each model.

The system must:

  • Add cache entries for accepted tokens.
  • Discard or roll back rejected speculative positions.
  • Keep tokenization aligned.
  • Avoid copying so much cache data that the optimization loses its benefit.

A separate draft model also consumes weights and memory. This can reduce the number of requests that fit on one accelerator.

Where it helps

Speculative decoding is most attractive when:

  • The target model is expensive.
  • Responses contain many output tokens.
  • The draft model is much cheaper.
  • Draft and target predictions agree often.
  • Decode latency matters more than prompt prefill.
  • The serving stack supports efficient verification and cache management.

It may help less when prompts are long and outputs are short, because prefill dominates. It may also help less at high batch sizes if the server already uses the accelerator efficiently.

Evaluating a deployment

Measure the full service, not just accepted tokens:

  • Time to first token.
  • Inter-token latency.
  • Output tokens per second.
  • End-to-end latency.
  • Acceptance rate and accepted tokens per target step.
  • Requests per second at realistic concurrency.
  • Accelerator memory use.
  • Cost per completed request.
  • Output distribution or task quality.

Test representative prompts, languages, temperatures, and output lengths. A benchmark with easy, predictable text can overstate gains for real workloads.

Common misconceptions

  • The draft model does not replace the target model.
  • Rejected tokens are not shown to the user.
  • A high acceptance rate does not guarantee lower cost if the draft is expensive.
  • Longer speculative blocks are not always better.
  • The optimization mainly targets decode; it does not remove prompt prefill.
  • Correct speculative sampling is more than comparing one confidence score.

The key idea

Speculative decoding trades cheap proposal work for fewer sequential target-model steps. It is useful when the draft is inexpensive, agreement is high, and the serving system can verify and manage cache state efficiently without changing the intended target behavior.

Saved in this browser only. Not synced to your account.