GPT and Claude can write an explanation, produce code, or carry on a conversation. Their text generation is usually explained with one central idea: predict the next token using the context so far, append it, then repeat. That is autoregressive generation.
The term sounds more complicated than the mechanism. Understanding it explains why answers stream gradually, why long responses take time, and why an early mistake can influence everything that follows. It also helps separate three questions that are often mixed together: how a model generates, how its neural network is built, and what it was trained to do.
What does “autoregressive” actually mean?
Auto means “self.” Regressive comes from statistical models that predict a value using earlier values. An autoregressive model predicts the next part of a sequence using earlier parts of that same sequence.
In a traditional time-series model, tomorrow’s temperature might be predicted using temperatures from previous days. A simple autoregressive statistical model combines earlier observations with learned coefficients and an error term. A language model applies the broader sequence idea to text, using a much more expressive neural network rather than a simple linear equation.
The pieces of text are tokens, which can be words, parts of words, punctuation, or other units chosen by the tokenizer. The next-token prediction is conditioned on the available prefix: the prompt and, once generation starts, the tokens already produced.
For readability, imagine the following words each occupy one token. Actual token boundaries may differ:
Context Next token chosen
"The cat" " sat"
"The cat sat" " on"
"The cat sat on" " the"
"The cat sat on the" " mat"
The important part is the dependency. The choice of on is made with sat already in the context. The choice of mat is made with the whole earlier prefix available. Each selected token helps determine the next prediction.
At each step the model produces scores for many possible next tokens, not just one answer. A decoding rule turns those scores into a choice. It might select the highest-probability token or sample from a probability distribution. Autoregressive does not mean deterministic, and it does not mean the model can see only the immediately previous token.
A sequence is built from conditional predictions
Suppose the prompt is The cat and the continuation consists of three illustrative tokens: sat, on, and the. Its probability is built from the conditional probabilities at successive steps:
P(" sat", " on", " the" | prompt)
= P(" sat" | prompt)
* P(" on" | prompt, " sat")
* P(" the" | prompt, " sat", " on")
The vertical bar means “given.” Each factor asks how likely the next token is, given everything before it. This is an application of the probability chain rule, not a claim that the tokens are independent.
A neural autoregressive language model learns to estimate these conditional distributions. During inference, the process of running a trained model, the decoding loop uses them to construct an actual continuation. Generation ends when the system reaches a stopping condition, such as an end-of-sequence token, a configured stop sequence, or an output-token limit.
Selecting the most likely token at every step does not necessarily produce the most likely complete sequence. A locally attractive choice can lead to a less likely continuation. Sampling, beam search, and other decoding strategies make different tradeoffs; the underlying autoregressive dependency remains.
Autoregressive does not mean training must be sequential
There is a useful distinction between learning from a known sequence and generating an unknown one.
During standard causal language-model training, the full example is already available. For a sequence such as The cat sat on the mat, the training targets are shifted by one token: the representation at The predicts cat, the representation at cat predicts sat, and so on.
A causal transformer can calculate the predictions for many positions in parallel. A causal attention mask prevents each position from accessing future tokens, even though those tokens are present elsewhere in the training example. This use of the correct earlier tokens is often called teacher forcing.
During generation, the correct continuation is not supplied. The system has to choose a token before the next ordinary decoding step can use it. That creates a sequential dependency across generated tokens.
Training: known prefix -> predict the supplied next-token target
Generation: current prefix -> choose a token -> extend prefix -> repeat
This difference helps explain error propagation. Training typically supplies correct prefixes, while generation may feed the model its own incorrect choices. If it invents a function name early in a code example, later tokens may build a plausible implementation around that nonexistent function.
Autoregressive is a generation rule, not an architecture
The LLM architecture describes the network’s components and how information moves through them. Autoregression describes a sequence dependency. These are different axes.
A recurrent neural network can generate autoregressively. So can a transformer. “Transformer” therefore does not automatically mean “autoregressive,” and “autoregressive” does not automatically mean “transformer.”
Three common transformer arrangements make the distinction clearer:
- Decoder-only: processes a prefix with causal attention and commonly generates the continuation autoregressively. GPT-style language models are the familiar example.
- Encoder-only: reads an input with bidirectional attention, allowing a token to use both left and right context. BERT is the classic example.
- Encoder-decoder: an encoder reads the source input, and a decoder generates a separate output. Standard encoder-decoder models such as T5 and BART still use autoregressive output decoding.
An encoder-decoder translator can read the whole source sentence before producing the first translated token. Its decoder then generates the translation one token at a time, conditioned on the source representation and the translated tokens already emitted. It is both encoder-decoder and autoregressive.
Similarly, a mixture-of-experts model routes computation through selected neural-network blocks. That describes how compute is allocated inside the network, not whether output tokens depend on earlier ones. A mixture-of-experts transformer can still generate autoregressively.
What are the alternatives, and how do they compare?
Not every model has the job of extending a prefix. Some build representations of an existing input; others generate a whole sequence through parallel prediction or repeated refinement. The following approaches are not all mutually exclusive categories.
Masked language models learn to recover hidden tokens from surrounding text. BERT can use both sides of a gap in The cat [MASK] on the mat. This is useful for learning representations that support classification, extraction, and ranking. Standard BERT is not a drop-in open-ended chat generator: recovering a missing token in an existing sentence is a different task from deciding an entire answer’s length and content.
Non-autoregressive sequence models aim to predict multiple output positions without the usual left-to-right dependency on previously emitted tokens. Some first predict the output length; others use latent variables or repeated refinement. They can expose more parallelism, but must still solve coordination problems such as word order, agreement, and avoiding repeated or missing content.
Diffusion and iterative-denoising models generate by gradually refining a noisy or incomplete state. In continuous image diffusion, that state may be noisy image data. In masked text diffusion, it may be a sequence with masked token positions. LLaDA is a published example of a language diffusion model that learns to recover masked tokens through a reverse generation process.
These methods can update several positions in a step and use information on both sides of an unresolved position. They still require computation across refinement steps. Some implementations also operate in blocks, combining sequential and parallel behavior. “Non-autoregressive” is not a promise of one-pass generation or universally lower latency.
| Approach | What a prediction uses | Typical strength | Important limitation |
|---|---|---|---|
| Autoregressive generation | Prompt plus earlier output tokens | Flexible variable-length text and natural prefix streaming | Ordinary decoding has a sequential output dependency |
| BERT-style masked modeling | Visible context around hidden positions | Representations for understanding and discriminative tasks | Not inherently an open-ended response generator |
| Non-autoregressive sequence generation | Source input and model-specific length or coordination signals | More output-position parallelism | Output positions still need coherent coordination |
| Text diffusion or iterative denoising | A partially resolved sequence, often with bidirectional context | Refinement of multiple positions and flexible generation orders | Multiple refinement steps; speed and quality depend on implementation |
There is no universal winner. Compare actual quality, latency, memory use, and supported tasks on the same workload rather than comparing only the category names.
What type are OpenAI’s models?
For GPT-family text generation, the practical mental model is autoregressive: later text is generated using the preceding context. The name GPT expands to Generative Pre-trained Transformer, but the name alone is not a complete architecture specification.
There is particularly clear public evidence for GPT-4. The GPT-4 technical report explicitly describes it as a transformer-style model pre-trained to predict the next token in a document. It also says that further architecture details, including model size, are not disclosed.
Older GPT models, GPT-4-generation models such as GPT-4o, and newer GPT-family assistants belong in the autoregressive text-generation discussion, not the BERT-style masked-encoder category. However, the GPT-4 report is not evidence for every implementation detail of every later release.
As of November 5, 2026, OpenAI’s model catalog highlights GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna. Their public documentation describes capabilities, input and output modalities, and serving features. It should not be read as a disclosure of their complete network topology, expert routing, parameter counts, or decoding implementation.
| OpenAI family or example | Useful classification for this article | What the label does not establish |
|---|---|---|
| GPT-3 and GPT-4 | Autoregressive transformer language models | The complete architecture of later releases |
| GPT-4o and later GPT-family text assistants, including the current GPT-6 lineup | Autoregressive text-generation mental model; multimodal capabilities depend on the specific model | An identical architecture across releases or output modalities |
| o-series reasoning models and newer reasoning-capable GPT models | Reasoning-oriented text generators, not a separate masked-language-model category | That “reasoning” replaces sequence generation with a disclosed new architecture |
OpenAI’s reasoning documentation explains that reasoning models use internal reasoning tokens before producing a response. Those tokens can add computation and latency beyond the visible answer. “Reasoning model” describes capabilities, training, and inference behavior; it is not the opposite of “autoregressive.”
Also distinguish ChatGPT, the product, from the underlying model. A ChatGPT response can involve model selection, tools, retrieval, and other orchestration. Classifying a text model does not classify every component of that product.
OpenAI also builds systems outside this text-generation category. For example, the published DALL-E 2 work uses diffusion components for image generation. Do not infer that every OpenAI image or video model uses the same method, or that every modality of a multimodal model follows the same text-token loop.
What type are Anthropic’s Claude models?
Claude’s text generation belongs in the same autoregressive discussion. Anthropic’s research on tracing model computations describes Claude producing text one piece at a time and studies the computations inside Claude 3.5 Haiku. The research also shows why next-token generation should not be confused with an inability to plan ahead.
The current Claude model overview, checked on November 5, 2026, lists Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, and Claude Haiku 5.5. These are Claude language-model offerings with different capability and latency tradeoffs, not names for masked, diffusion, or encoder-decoder categories.
| Anthropic offering | Useful classification for this article | Main distinction in the current catalog |
|---|---|---|
| Claude Fable | Autoregressive text-generation mental model | Demanding reasoning and long-horizon agentic work |
| Claude Opus | Autoregressive text-generation mental model | Long-running agentic coding and knowledge work |
| Claude Sonnet | Autoregressive text-generation mental model | Balance of speed and intelligence |
| Claude Haiku | Autoregressive text-generation mental model | High-volume, latency-sensitive workloads |
Anthropic’s thinking documentation describes reasoning content before the final answer and, when tools are involved, between tool calls. Thinking changes how computation is spent; it does not make Claude a BERT-style masked encoder or establish a diffusion architecture.
Claude is commonly discussed as a transformer-based generative language-model family. Public research supports that broad picture for the studied releases. It does not justify inventing the exact topology, parameter count, dense-versus-expert design, or multimodal internals of every current Claude model.
For both providers, keep two confidence levels separate: the autoregressive text-generation model is useful and well established; a complete architecture specification for every proprietary release is not public. Model catalogs change, so consult the linked official pages when choosing an API model.
Does predicting the next token mean it cannot reason?
No. A training objective tells us what prediction is rewarded, not everything the network can learn while solving that prediction task.
Predicting useful continuations across mathematics, code, explanations, and conversations can require representations of facts, relationships, syntax, and multi-step procedures. Generating tokens in order does not prohibit the network from representing a future goal.
Anthropic’s interpretability research provides a concrete example: in studied poetry tasks, Claude represented possible rhyming endpoints before generating the words leading to them. That is compatible with autoregression because future output tokens were not supplied to the model. It formed an internal plan and then generated text sequentially.
The opposite exaggeration is also a mistake. Autoregressive generation does not guarantee correct reasoning, verified facts, or faithful explanations of internal computation. A model can generate a convincing argument for an incorrect answer. Validate important claims and use executable checks for code and calculations.
What this means when building an application
The sequence dependency has practical consequences:
Separate prompt processing from output generation. The initial prompt-processing stage, often called prefill, can process many prompt positions in parallel. Ordinary decoding then advances through generated tokens. Measure time to first token and time to complete the response separately.
Control output length. A short validated answer often costs less and arrives sooner than a long explanation. Reasoning tokens, tool calls, and network overhead also matter, so visible answer length alone is not a complete latency estimate.
Treat streaming as partial output. Seeing text appear early improves perceived responsiveness, but a streamed fragment may not yet be valid JSON or a complete instruction. Delay consequential actions until the required output is complete and validated.
Watch early commitments. An invented identifier or incorrect assumption can become part of the prefix for later predictions. Supply exact identifiers, ground factual tasks in evidence, and check generated artifacts rather than trusting fluent continuation.
Understand what acceleration changes. A key-value cache reuses earlier attention computations instead of recomputing the whole prefix from scratch. Speculative decoding can propose several tokens and let a target model verify them together. These techniques can improve serving speed without abandoning the autoregressive distribution; actual gains depend on the model and workload.
Choose the model for the task. Use a generative assistant for flexible writing, dialogue, and code. Consider an encoder, embedding model, or reranker when the real requirement is representing or scoring existing text. Evaluate alternative generators when parallel refinement or infilling is central to the workflow.
The key idea
An autoregressive model predicts each next piece using the sequence available so far. It does not necessarily use a simple statistical regression, see only one earlier token, or lack an internal plan.
GPT and Claude are best understood through this autoregressive text-generation mental model, with different capabilities and largely private implementation details. BERT-style masked models, encoder-decoder systems, and diffusion generators explain other design choices, but the categories overlap: an encoder-decoder model can still generate autoregressively, and a transformer can support non-autoregressive generation.
The engineering question is not just “What type is this model?” It is “What dependency does its output have, what task does it support, and what evidence do I have about its implementation?”
Sources and further reading
- OpenAI: GPT-4 Technical Report
- OpenAI: Model catalog
- OpenAI: Reasoning models
- Anthropic: Claude models overview
- Anthropic: Tracing the thoughts of a large language model
- Anthropic: Thinking
- Devlin et al.: BERT
- Raffel et al.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Nie et al.: Large Language Diffusion Models
- Ramesh et al.: Hierarchical Text-Conditional Image Generation with CLIP Latents (DALL-E 2)