HQAI Engineering HQ
Search
Checking sign-in...

What Are Autoregressive Models? How GPT and Claude Generate Text

Article19 min readAI Engineering HQ

GPT and Claude can write an explanation, produce code, or carry on a conversation. Their text generation is usually explained with one central idea: predict the next token using the context so far, append it, then repeat. That is autoregressive generation.

The term sounds more complicated than the mechanism. Understanding it explains why answers stream gradually, why long responses take time, and why an early mistake can influence everything that follows. It also helps separate three questions that are often mixed together: how a model generates, how its neural network is built, and what it was trained to do.

What does “autoregressive” actually mean?

Auto means “self.” Regressive comes from statistical models that predict a value using earlier values. An autoregressive model predicts the next part of a sequence using earlier parts of that same sequence.

In a traditional time-series model, tomorrow’s temperature might be predicted using temperatures from previous days. A simple autoregressive statistical model combines earlier observations with learned coefficients and an error term. A language model applies the broader sequence idea to text, using a much more expressive neural network rather than a simple linear equation.

The pieces of text are tokens, which can be words, parts of words, punctuation, or other units chosen by the tokenizer. The next-token prediction is conditioned on the available prefix: the prompt and, once generation starts, the tokens already produced.

For readability, imagine the following words each occupy one token. Actual token boundaries may differ:

Context                         Next token chosen
"The cat"                       " sat"
"The cat sat"                   " on"
"The cat sat on"                " the"
"The cat sat on the"            " mat"

The important part is the dependency. The choice of on is made with sat already in the context. The choice of mat is made with the whole earlier prefix available. Each selected token helps determine the next prediction.

At each step the model produces scores for many possible next tokens, not just one answer. A decoding rule turns those scores into a choice. It might select the highest-probability token or sample from a probability distribution. Autoregressive does not mean deterministic, and it does not mean the model can see only the immediately previous token.

The autoregressive text generation loop The current prefix enters the model, which scores possible next tokens. A decoding rule chooses one token, which is appended to the prefix. Unless a stopping condition is met, the extended prefix feeds the next prediction. The cat and sat are illustrative text, not fixed tokenizer boundaries. Current prefix "The cat" Run the model score next tokens Decode choose " sat" Append token "The cat sat" Continue? Use the extended prefix for the next step. Stop when a stopping condition is met.
The output token becomes input to the next prediction; this feedback dependency is what makes generation autoregressive.

A sequence is built from conditional predictions

Suppose the prompt is The cat and the continuation consists of three illustrative tokens: sat, on, and the. Its probability is built from the conditional probabilities at successive steps:

P(" sat", " on", " the" | prompt)
  = P(" sat" | prompt)
  * P(" on"  | prompt, " sat")
  * P(" the" | prompt, " sat", " on")

The vertical bar means “given.” Each factor asks how likely the next token is, given everything before it. This is an application of the probability chain rule, not a claim that the tokens are independent.

A neural autoregressive language model learns to estimate these conditional distributions. During inference, the process of running a trained model, the decoding loop uses them to construct an actual continuation. Generation ends when the system reaches a stopping condition, such as an end-of-sequence token, a configured stop sequence, or an output-token limit.

Selecting the most likely token at every step does not necessarily produce the most likely complete sequence. A locally attractive choice can lead to a less likely continuation. Sampling, beam search, and other decoding strategies make different tradeoffs; the underlying autoregressive dependency remains.

Autoregressive does not mean training must be sequential

There is a useful distinction between learning from a known sequence and generating an unknown one.

During standard causal language-model training, the full example is already available. For a sequence such as The cat sat on the mat, the training targets are shifted by one token: the representation at The predicts cat, the representation at cat predicts sat, and so on.

A causal transformer can calculate the predictions for many positions in parallel. A causal attention mask prevents each position from accessing future tokens, even though those tokens are present elsewhere in the training example. This use of the correct earlier tokens is often called teacher forcing.

During generation, the correct continuation is not supplied. The system has to choose a token before the next ordinary decoding step can use it. That creates a sequential dependency across generated tokens.

Training:    known prefix -> predict the supplied next-token target
Generation: current prefix -> choose a token -> extend prefix -> repeat
Parallel training predictions versus sequential generation In training, the known prefixes The, The cat, and The cat sat predict supplied targets cat, sat, and on. These predictions can be calculated in parallel with a causal mask hiding future positions. In generation, a prefix produces sat, and only after that choice can the extended prefix produce on. Words stand in for tokens in this simplified example. TRAINING: the correct sequence is already supplied Prefix: The Target: cat Prefix: The cat Target: sat Prefix: The cat sat Target: on Parallel predictions; each position is prevented from seeing its target or later tokens. GENERATION: later predictions use earlier choices Prefix: The cat Choose: sat Prefix: The cat sat Choose: on Continue from The cat sat on
Known training targets allow parallel prediction without future-token access; ordinary generation must first supply its own next token.

This difference helps explain error propagation. Training typically supplies correct prefixes, while generation may feed the model its own incorrect choices. If it invents a function name early in a code example, later tokens may build a plausible implementation around that nonexistent function.

Autoregressive is a generation rule, not an architecture

The LLM architecture describes the network’s components and how information moves through them. Autoregression describes a sequence dependency. These are different axes.

A recurrent neural network can generate autoregressively. So can a transformer. “Transformer” therefore does not automatically mean “autoregressive,” and “autoregressive” does not automatically mean “transformer.”

Three common transformer arrangements make the distinction clearer:

  • Decoder-only: processes a prefix with causal attention and commonly generates the continuation autoregressively. GPT-style language models are the familiar example.
  • Encoder-only: reads an input with bidirectional attention, allowing a token to use both left and right context. BERT is the classic example.
  • Encoder-decoder: an encoder reads the source input, and a decoder generates a separate output. Standard encoder-decoder models such as T5 and BART still use autoregressive output decoding.

An encoder-decoder translator can read the whole source sentence before producing the first translated token. Its decoder then generates the translation one token at a time, conditioned on the source representation and the translated tokens already emitted. It is both encoder-decoder and autoregressive.

An encoder-decoder can still generate autoregressively An encoder reads the complete source input and supplies contextual representations to the decoder. The decoder also receives the output prefix, chooses the next output token, and feeds the extended output prefix back for another step. Reading the full source does not remove the dependency between generated output tokens. Encoder reads the complete source source representations Output prefix tokens generated so far Decoder choose next output token Append to output extend the output prefix Repeat output decoding until a stopping condition.
The encoder provides source context, while the decoder still builds its output autoregressively from the tokens emitted so far.

Similarly, a mixture-of-experts model routes computation through selected neural-network blocks. That describes how compute is allocated inside the network, not whether output tokens depend on earlier ones. A mixture-of-experts transformer can still generate autoregressively.

What are the alternatives, and how do they compare?

Not every model has the job of extending a prefix. Some build representations of an existing input; others generate a whole sequence through parallel prediction or repeated refinement. The following approaches are not all mutually exclusive categories.

Masked language models learn to recover hidden tokens from surrounding text. BERT can use both sides of a gap in The cat [MASK] on the mat. This is useful for learning representations that support classification, extraction, and ranking. Standard BERT is not a drop-in open-ended chat generator: recovering a missing token in an existing sentence is a different task from deciding an entire answer’s length and content.

Non-autoregressive sequence models aim to predict multiple output positions without the usual left-to-right dependency on previously emitted tokens. Some first predict the output length; others use latent variables or repeated refinement. They can expose more parallelism, but must still solve coordination problems such as word order, agreement, and avoiding repeated or missing content.

Diffusion and iterative-denoising models generate by gradually refining a noisy or incomplete state. In continuous image diffusion, that state may be noisy image data. In masked text diffusion, it may be a sequence with masked token positions. LLaDA is a published example of a language diffusion model that learns to recover masked tokens through a reverse generation process.

These methods can update several positions in a step and use information on both sides of an unresolved position. They still require computation across refinement steps. Some implementations also operate in blocks, combining sequential and parallel behavior. “Non-autoregressive” is not a promise of one-pass generation or universally lower latency.

Extending a prefix, filling a gap, and refining a sequence Three simplified examples show different prediction tasks. Autoregressive generation extends The cat to The cat sat and then The cat sat on. Masked modeling uses The cat and on the mat around a mask to predict sat. Iterative masked generation resolves two of four masked positions as cat and on, then resolves the other two as The and sat. The examples illustrate dependencies, not actual tokenizer boundaries or a specific diffusion schedule. AUTOREGRESSIVE: extend the prefix The cat The cat sat The cat sat on Each newly chosen token is available to the next prediction. MASKED MODELING: recover a gap using both sides The cat [MASK] on the mat Predict the missing token: sat Surrounding text is supplied; this is not inherently open-ended generation. ITERATIVE MASKED GENERATION: resolve several positions over steps [M] [M] [M] [M] [M] cat [M] on The cat sat on [M] means masked; this is an illustrative refinement order, not a measured schedule.
Prefix extension, gap recovery, and iterative generation expose different information at each step; words and refinement stages are simplified.
Approach What a prediction uses Typical strength Important limitation
Autoregressive generation Prompt plus earlier output tokens Flexible variable-length text and natural prefix streaming Ordinary decoding has a sequential output dependency
BERT-style masked modeling Visible context around hidden positions Representations for understanding and discriminative tasks Not inherently an open-ended response generator
Non-autoregressive sequence generation Source input and model-specific length or coordination signals More output-position parallelism Output positions still need coherent coordination
Text diffusion or iterative denoising A partially resolved sequence, often with bidirectional context Refinement of multiple positions and flexible generation orders Multiple refinement steps; speed and quality depend on implementation

There is no universal winner. Compare actual quality, latency, memory use, and supported tasks on the same workload rather than comparing only the category names.

What type are OpenAI’s models?

For GPT-family text generation, the practical mental model is autoregressive: later text is generated using the preceding context. The name GPT expands to Generative Pre-trained Transformer, but the name alone is not a complete architecture specification.

There is particularly clear public evidence for GPT-4. The GPT-4 technical report explicitly describes it as a transformer-style model pre-trained to predict the next token in a document. It also says that further architecture details, including model size, are not disclosed.

Older GPT models, GPT-4-generation models such as GPT-4o, and newer GPT-family assistants belong in the autoregressive text-generation discussion, not the BERT-style masked-encoder category. However, the GPT-4 report is not evidence for every implementation detail of every later release.

As of November 5, 2026, OpenAI’s model catalog highlights GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna. Their public documentation describes capabilities, input and output modalities, and serving features. It should not be read as a disclosure of their complete network topology, expert routing, parameter counts, or decoding implementation.

OpenAI family or example Useful classification for this article What the label does not establish
GPT-3 and GPT-4 Autoregressive transformer language models The complete architecture of later releases
GPT-4o and later GPT-family text assistants, including the current GPT-6 lineup Autoregressive text-generation mental model; multimodal capabilities depend on the specific model An identical architecture across releases or output modalities
o-series reasoning models and newer reasoning-capable GPT models Reasoning-oriented text generators, not a separate masked-language-model category That “reasoning” replaces sequence generation with a disclosed new architecture

OpenAI’s reasoning documentation explains that reasoning models use internal reasoning tokens before producing a response. Those tokens can add computation and latency beyond the visible answer. “Reasoning model” describes capabilities, training, and inference behavior; it is not the opposite of “autoregressive.”

Also distinguish ChatGPT, the product, from the underlying model. A ChatGPT response can involve model selection, tools, retrieval, and other orchestration. Classifying a text model does not classify every component of that product.

OpenAI also builds systems outside this text-generation category. For example, the published DALL-E 2 work uses diffusion components for image generation. Do not infer that every OpenAI image or video model uses the same method, or that every modality of a multimodal model follows the same text-token loop.

What type are Anthropic’s Claude models?

Claude’s text generation belongs in the same autoregressive discussion. Anthropic’s research on tracing model computations describes Claude producing text one piece at a time and studies the computations inside Claude 3.5 Haiku. The research also shows why next-token generation should not be confused with an inability to plan ahead.

The current Claude model overview, checked on November 5, 2026, lists Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5. These are Claude language-model offerings with different capability and latency tradeoffs, not names for masked, diffusion, or encoder-decoder categories.

Anthropic offering Useful classification for this article Main distinction in the current catalog
Claude Fable Autoregressive text-generation mental model Demanding reasoning and long-horizon agentic work
Claude Opus Autoregressive text-generation mental model Long-running agentic coding and knowledge work
Claude Sonnet Autoregressive text-generation mental model Balance of speed and intelligence
Claude Haiku Autoregressive text-generation mental model High-volume, latency-sensitive workloads

Anthropic’s thinking documentation describes reasoning content before the final answer and, when tools are involved, between tool calls. Thinking changes how computation is spent; it does not make Claude a BERT-style masked encoder or establish a diffusion architecture.

Claude is commonly discussed as a transformer-based generative language-model family. Public research supports that broad picture for the studied releases. It does not justify inventing the exact topology, parameter count, dense-versus-expert design, or multimodal internals of every current Claude model.

For both providers, keep two confidence levels separate: the autoregressive text-generation model is useful and well established; a complete architecture specification for every proprietary release is not public. Model catalogs change, so consult the linked official pages when choosing an API model.

Does predicting the next token mean it cannot reason?

No. A training objective tells us what prediction is rewarded, not everything the network can learn while solving that prediction task.

Predicting useful continuations across mathematics, code, explanations, and conversations can require representations of facts, relationships, syntax, and multi-step procedures. Generating tokens in order does not prohibit the network from representing a future goal.

Anthropic’s interpretability research provides a concrete example: in studied poetry tasks, Claude represented possible rhyming endpoints before generating the words leading to them. That is compatible with autoregression because future output tokens were not supplied to the model. It formed an internal plan and then generated text sequentially.

The opposite exaggeration is also a mistake. Autoregressive generation does not guarantee correct reasoning, verified facts, or faithful explanations of internal computation. A model can generate a convincing argument for an incorrect answer. Validate important claims and use executable checks for code and calculations.

What this means when building an application

The sequence dependency has practical consequences:

Separate prompt processing from output generation. The initial prompt-processing stage, often called prefill, can process many prompt positions in parallel. Ordinary decoding then advances through generated tokens. Measure time to first token and time to complete the response separately.

Control output length. A short validated answer often costs less and arrives sooner than a long explanation. Reasoning tokens, tool calls, and network overhead also matter, so visible answer length alone is not a complete latency estimate.

Treat streaming as partial output. Seeing text appear early improves perceived responsiveness, but a streamed fragment may not yet be valid JSON or a complete instruction. Delay consequential actions until the required output is complete and validated.

Watch early commitments. An invented identifier or incorrect assumption can become part of the prefix for later predictions. Supply exact identifiers, ground factual tasks in evidence, and check generated artifacts rather than trusting fluent continuation.

Understand what acceleration changes. A key-value cache reuses earlier attention computations instead of recomputing the whole prefix from scratch. Speculative decoding can propose several tokens and let a target model verify them together. These techniques can improve serving speed without abandoning the autoregressive distribution; actual gains depend on the model and workload.

Choose the model for the task. Use a generative assistant for flexible writing, dialogue, and code. Consider an encoder, embedding model, or reranker when the real requirement is representing or scoring existing text. Evaluate alternative generators when parallel refinement or infilling is central to the workflow.

The key idea

An autoregressive model predicts each next piece using the sequence available so far. It does not necessarily use a simple statistical regression, see only one earlier token, or lack an internal plan.

GPT and Claude are best understood through this autoregressive text-generation mental model, with different capabilities and largely private implementation details. BERT-style masked models, encoder-decoder systems, and diffusion generators explain other design choices, but the categories overlap: an encoder-decoder model can still generate autoregressively, and a transformer can support non-autoregressive generation.

The engineering question is not just “What type is this model?” It is “What dependency does its output have, what task does it support, and what evidence do I have about its implementation?”

Sources and further reading

Saved in this browser only. Not synced to your account.