How large language models actually work — a plain-English guide for developers

Large language models (LLMs) are behind ChatGPT, Claude, Gemini and the AI features now appearing in almost every product. Most people use them every day. Far fewer can explain what actually happens between typing a question and reading the answer.

That gap matters. If you understand the mechanics, you write better prompts, you know when to trust an answer and when to check it, and you design better software around these models. This guide walks through the whole pipeline — from text to tokens to numbers to a reply — in plain English.

The one-sentence version

An LLM is a very large neural network trained to do one thing: given some text, predict what comes next. Everything else — answering questions, writing code, summarizing documents — emerges from doing that single task extremely well, one small piece at a time.

When you send a prompt, the model predicts the most likely next piece of text, appends it, and repeats. A 500-word answer is a few hundred of these predictions in a row.

Step 1: Text becomes tokens

Models don’t read letters or whole words. They read tokens — chunks of text that a tokenizer has learned are useful building blocks. A token is often a whole common word, part of a longer word, a space plus a word, or a punctuation mark.

For example, a sentence like this might be split as follows (the exact split depends on the model’s tokenizer):

"Understanding tokenization matters."
→ ["Under", "standing", " token", "ization", " matters", "."]

Most modern tokenizers use a technique called byte-pair encoding (BPE) or a close relative. It starts from individual bytes and repeatedly merges the pairs that appear together most often in a huge training corpus. Common words end up as a single token; rare words get split into several.

This has practical consequences you have probably noticed:

• Pricing and limits are counted in tokens, not words. In English, a token is roughly three-quarters of a word on average; other languages and code often use more tokens per word.
• Models can struggle with character-level tasks — counting letters in a word, reversing a string — because they never see individual characters, only tokens.
• Unusual spellings, rare names and some languages cost more tokens and are often handled less reliably.

Step 2: Tokens become vectors (embeddings)

Each token is mapped to an ID, and each ID is mapped to a long list of numbers called an embedding — typically hundreds or thousands of numbers. You can think of an embedding as a point in a very high-dimensional space.

During training, the model learns to place tokens with related meanings near each other. “Doctor” ends up closer to “nurse” than to “bicycle”. Directions in this space come to capture relationships: tense, plurality, topic, tone.

The model also needs to know the order of tokens — “dog bites man” is not “man bites dog”. So position information is added to each token’s vector. Modern models use techniques such as rotary position embeddings, but the idea is the same: every vector carries both what the token is and where it sits.

Step 3: The transformer and attention

The heart of every modern LLM is the transformer architecture, introduced in the 2017 paper Attention Is All You Need. A transformer is a stack of identical layers — dozens of them in a large model. Each layer refines every token’s vector using two main components.

Self-attention: tokens looking at each other

The key idea is self-attention. For every token, the layer asks: which other tokens in this text matter for understanding me, and how much?

Take the sentence: “The trophy didn’t fit in the suitcase because it was too big.” To understand what “it” refers to, the model needs to connect “it” with “trophy” and with “big”. Attention is the mechanism that lets the vector for “it” pull in information from those tokens.

Mechanically, each token’s vector is turned into three smaller vectors:

• a query — what this token is looking for,
• a key — what this token offers to others,
• a value — the information it will pass along if selected.

The model compares each token’s query with every other token’s key. High similarity means high attention. It then takes a weighted mix of the values. The result is a new vector for each token that blends in context from the rest of the text.

In a language model, attention is causal: a token can only look at tokens before it, never after. That’s what makes next-token prediction an honest task — the model can’t peek at the answer.

Transformers run many attention calculations in parallel, called attention heads. Different heads learn to track different kinds of relationships — grammar, references, formatting, code structure — although in practice their roles are learned, not designed.

Feed-forward layers: thinking about each token

After attention, each token’s vector passes through a feed-forward network — a block of learned weights applied to each position independently. Research suggests a lot of factual association lives in these layers: the patterns that link “the capital of France” to “Paris”.

Stack attention and feed-forward blocks dozens of times and you get vectors that encode remarkably rich information about each position in the text.

Step 4: From vectors to the next token

After the final layer, the vector at the last position is converted into a score for every token in the vocabulary — often 100,000 or more possible tokens. Those scores are turned into probabilities.

So the model’s real output isn’t a word. It’s a probability distribution, something like:

"The capital of France is"
  " Paris"   0.91
  " a"       0.03
  " the"     0.02
  " located" 0.01
  ...

Step 5: Sampling — choosing what to say

Something has to pick one token from that distribution. That’s sampling, and it is controlled by settings you may have seen in an API:

• Temperature sharpens or flattens the distribution. Low temperature (close to 0) makes the model pick the most likely token almost every time — good for extraction, classification and code. Higher temperature lets less likely tokens through — more varied, sometimes more creative, sometimes less accurate.
• Top-p (nucleus sampling) only considers the smallest set of tokens whose probabilities add up to p, cutting off the long tail of unlikely options.
• Max tokens caps the length of the reply.

The chosen token is appended to the text and the whole process runs again for the next token. This loop is why answers stream to your screen word by word — they are literally being produced that way.

To make this fast, models cache the keys and values they have already computed for earlier tokens (the KV cache), so each new step only has to process one new token.

How a model learns all this: training

None of the behavior above is programmed by hand. The billions of numbers inside the model — its parameters or weights — start out random and are adjusted through training.

Pre-training: predicting the internet

In pre-training, the model reads an enormous amount of text: web pages, books, code, papers. At every position it predicts the next token, compares its prediction with the real one, and nudges all of its weights slightly to make the correct token more likely next time. This is done trillions of times on large clusters of GPUs or other accelerators.

To get good at predicting the next token across that much text, the model is pushed to pick up grammar, facts, writing styles, reasoning patterns and programming languages — because all of those help with prediction.

A pre-trained “base model” is powerful but awkward. Ask it a question and it might continue with three more questions, because that’s what text on the internet often does.

Post-training: becoming an assistant

To turn a base model into a helpful assistant, developers run further training stages:

• Supervised fine-tuning (instruction tuning): training on many examples of instructions paired with high-quality responses, so the model learns the assistant format.
• Learning from feedback: methods such as reinforcement learning from human feedback (RLHF), AI-assisted feedback, and preference optimization. People (or models guided by written principles) compare responses, and the model is trained to prefer the better ones — more helpful, more honest, safer.
• Training for reasoning and tool use: many recent models are also trained to work through problems step by step before answering and to call tools such as search or code execution.

The context window: the model’s working memory

An LLM has no memory between conversations by default. Everything it knows about your situation must be inside the context window — the text it is given for this request: the system prompt, the conversation so far, any documents you included and its own reply so far.

Context windows have grown from a few thousand tokens to hundreds of thousands or more. But bigger isn’t automatically better: models can pay less attention to details buried in the middle of very long inputs, and every token costs time and money. When an app seems to “remember” you, it is usually storing notes somewhere and putting them back into the context.

Why models hallucinate

A model produces the most plausible continuation, not a looked-up fact. Usually plausible and true overlap. When the model lacks the knowledge — a niche topic, a recent event, a specific number — the most plausible-sounding continuation can be confidently wrong. That’s a hallucination.

The practical fixes all come from the mechanics above:

• Give it the facts. Put the relevant documents into the context (this is what retrieval-augmented generation, or RAG, does).
• Let it use tools. Search, databases and calculators beat memory for exact facts and arithmetic.
• Ask for sources and permission to say “I don’t know”, then check what matters.
• Lower the temperature for tasks that need precision.

What LLMs are — and aren’t

It is easy to overstate or understate these systems. A balanced view:

• They are not databases. Knowledge is stored diffusely in weights and can be out of date or wrong.
• They are not just autocomplete in the dismissive sense. Predicting text well at this scale requires internal representations that support real reasoning, translation, summarization and code generation.
• Their knowledge has a cut-off date — the end of their training data — unless you give them newer information.
• They are sensitive to wording and context, which is why prompt design, examples and clear instructions make such a difference.

Why this matters if you build software

Understanding the pipeline turns vague advice into concrete decisions:

• Tokens explain cost and latency. Shorter prompts and outputs are cheaper and faster.
• The context window explains why you design what goes into each request instead of hoping the model “knows”.
• Sampling explains why outputs vary — and when to set temperature low for predictable results.
• Training explains why models follow clear instructions and examples so well — that is what post-training rewarded.
• Hallucinations explain why production AI features need grounding, validation and a human-friendly fallback.

LLMs are a new kind of building block. You don’t need to train one to use them well — but knowing how they work is what separates a fragile demo from a reliable feature.

Key terms at a glance

• Token: a chunk of text the model reads and writes; the unit of pricing and limits.
• Embedding: a list of numbers representing a token’s meaning.
• Transformer: the layered neural-network architecture used by modern LLMs.
• Attention: the mechanism that lets each token draw information from earlier tokens.
• Parameters / weights: the learned numbers inside the model.
• Pre-training: learning to predict the next token on a huge corpus.
• Fine-tuning / RLHF: further training that makes a model a helpful, safe assistant.
• Context window: the maximum text the model can consider in one request.
• Temperature: how adventurous sampling is.
• Hallucination: a fluent but false output.