How Do Large Language Models Work? A Plain-English Guide

Large language models work by doing one thing over and over: predicting the next small chunk of text, called a token, based on everything that came before it. Train that prediction engine on a huge portion of the internet, stack enough computing power underneath it, then teach it to follow instructions, and you get ChatGPT, Claude, and Gemini. Everything these systems do, from writing code to passing bar exams, emerges from that single trick repeated billions of times.

That answer is accurate but compressed, and the compression hides the interesting parts: how “predict the next word” turns into apparent reasoning, why these models confidently make things up, and why they can write a sonnet but fail at counting the letters in “strawberry.” This guide unpacks the whole pipeline in plain English. No math, no code, and no hand-waving about “thinking machines” either.

Table of contents

• The one-sentence version, expanded
• Tokens: how models read
• Embeddings: turning words into meaning
• The transformer: attention is the engine
• How LLMs are trained
• What happens when you hit send
• Why LLMs hallucinate
• Context windows: the model’s working memory
• Reasoning models: teaching LLMs to think longer
• Model sizes, mixture of experts, and open weights
• What LLMs still can’t do
• What this means for how you use them
• Key takeaways
• Frequently asked questions

The one-sentence version, expanded

An LLM is a neural network, a piece of software loosely inspired by how neurons connect, that has been adjusted, parameter by parameter, until it’s extremely good at guessing what text comes next. The models behind ChatGPT and Claude have hundreds of billions of parameters: numerical dials that were tuned during training and now encode everything the model “knows.”

Here’s the part people find hard to accept: there’s no database of facts inside, no rulebook of grammar, no dictionary. Ask a model for the capital of France and it doesn’t look anything up. It produces “Paris” because, across the trillions of words it trained on, sequences about capitals of France overwhelmingly continue that way. Knowledge, in an LLM, is a pattern in the dials, not an entry in a table.

The surprise of the last few years, and the reason you’re reading about this at all, is how much capability falls out of that setup at scale. Predicting text well turns out to require modeling the things the text is about. To continue “the trophy didn’t fit in the suitcase because it was too big,” you need something like an understanding that trophies go inside suitcases and “it” refers to the trophy. Researchers argue, sincerely, about whether to call this understanding. What’s not in dispute is that it works well enough to be useful, and fails in ways worth understanding before you rely on it.

Tokens: how models read

Models don’t see letters or words. Before your prompt reaches the model, a tokenizer chops it into tokens: chunks that are sometimes whole words, sometimes fragments. “Cat” is one token. “Transformers” might be split into “transform” and “ers.” A rough rule for English is that a token averages about three quarters of a word, so 1,000 tokens is roughly 750 words.

Tokenization explains several everyday oddities. Why do models fumble questions like “how many r’s are in strawberry”? Because the model never sees the letters; it sees one or two token IDs, and the spelling inside them isn’t directly visible. Why is AI often worse in Thai or Amharic than English? Partly because tokenizers were built around English-heavy data, so other languages get chopped into more, smaller pieces, which costs quality and money. Why does your API bill count tokens? Because tokens are the model’s actual unit of work: every one processed and generated has a compute cost.

You can see tokenization yourself with OpenAI’s free tokenizer tool, which is worth thirty seconds of playing with; watching “unbelievable” split into pieces makes the whole concept click.

Embeddings: turning words into meaning

Each token gets converted into an embedding: a long list of numbers, thousands of them, that acts as coordinates in a kind of meaning space. Tokens used in similar ways end up near each other. “King” sits near “queen,” “doctor” near “nurse,” “Paris” near “London.”

Nobody designed this map. It emerges during training because placing related words near each other makes prediction more accurate. The famous party trick is that directions in this space carry meaning: the arrow from “king” to “queen” points roughly the same way as the arrow from “man” to “woman.” Meaning becomes geometry.

Embeddings also matter outside chatbots. When an AI-powered search feature finds relevant documents even though you didn’t use the exact keywords, that’s embedding similarity at work: your query and the document sit near each other in meaning space. It’s the technology behind the retrieval systems in our guide to building a private AI knowledge base.

The transformer: attention is the engine

The transformer is the architecture nearly every modern LLM uses, introduced by Google researchers in the 2017 paper “Attention Is All You Need.” The T in GPT stands for it. Its central mechanism, attention, is what made today’s models possible.

The problem attention solves: meaning depends on context, often distant context. In “the bank approved the loan” versus “the bank of the river flooded,” the word “bank” means different things, and the deciding clue can sit anywhere in the sentence, or three paragraphs back. Older architectures read text one word at a time, passing along a running summary, and by word two hundred they’d effectively forgotten word five.

Attention lets every token look directly at every other token and decide which ones matter for interpreting it. Processing “it” in “the trophy didn’t fit in the suitcase because it was too big,” attention lets “it” weigh its connection to “trophy” and “suitcase” and settle on the right one. Each token effectively asks the whole sequence “who’s relevant to me?” and blends the answers into its own representation.

A model stacks dozens of attention layers, and each layer refines things further. Researchers peering inside find lower layers handling grammar and structure while higher ones track meaning, relationships, and long-range plot. One more property mattered as much as the cleverness: attention runs on all tokens in parallel, which suits GPUs perfectly. Transformers won partly because they made better use of NVIDIA’s hardware than anything before them, and scale did the rest.

How LLMs are trained

Training happens in stages, and the stages explain a lot about model behavior.

Pretraining is the expensive part. The model chews through a filtered mix of web text, books, code, and papers, guessing the next token and getting nudged toward the right answer, trillions of times. This is where the hundreds of millions of dollars in compute go, and it’s why only a handful of organizations train frontier models from scratch.

A pretrained “base” model is capable but feral. Ask it a question and it might continue with more questions, because on the internet, questions often appear in lists of questions. Supervised fine-tuning fixes this by training on curated examples of good assistant behavior: question, then helpful answer, thousands of times, until the model adopts the persona.

The third stage is where “how it behaves” gets shaped. In reinforcement learning from human feedback (RLHF), people rank alternative model responses, and the model is adjusted toward what raters prefer. Anthropic layers on a variant called Constitutional AI, where a model critiques its own outputs against written principles. This stage is the source of both the polish and some of the annoyances: models that over-apologize, hedge, or flatter are reflecting what scored well with human raters, not laws of nature.

One consequence worth internalizing: a model’s knowledge freezes at pretraining. That’s the “knowledge cutoff,” and it’s why serious chat products now bolt on web search to cover recent events. The model isn’t remembering yesterday’s news; it’s reading search results you didn’t see and summarizing them.

What happens when you hit send

Generation, called inference, is the same loop every time. Your conversation is tokenized, the model processes it through its layers, and out comes a probability for every token in its vocabulary as the possible next one. After “The capital of France is,” the token “Paris” might carry most of the probability, with “located” and “a” holding slivers.

The model then samples from those probabilities rather than always taking the top pick. How adventurously it samples is controlled by a setting called temperature: low temperature sticks to the safest choices, higher temperature spreads the bets. This is why you get a different answer when you regenerate, and it’s a setting, not a bug. Deterministic sameness makes creative writing dull; looseness makes factual extraction unreliable. API users tune it per task.

The chosen token gets appended and the loop runs again, one token at a time, until a stop condition. That typewriter effect when ChatGPT streams its answer? Not theater. You’re watching the loop run in real time.

Notice what’s absent from the loop: checking. Nothing verifies a claim before emitting it. Any fact-checking has to come from outside the loop, whether that’s retrieval, tools, or you.

Why LLMs hallucinate

Hallucination, the confident fabrication of false information, isn’t a glitch bolted onto an otherwise truthful system. It’s the default behavior of a next-token predictor, for three reinforcing reasons.

The objective is plausibility, not truth. The model was trained to produce likely text. Most of the time, likely and true coincide, because the training data mostly says true things about well-covered topics. At the edges of its knowledge, though, likely text is whatever sounds right, and the model generates a plausible-sounding citation or biography with the same fluent confidence as a real one. Fluency is exactly what makes it dangerous: the errors don’t look like errors.

Knowledge is compressed, not stored. Hundreds of billions of parameters sounds like a lot until you compare it to the training data. Facts seen thousands of times survive compression well; things mentioned twice blur together. Ask about a famous ruling and you’ll do fine. Ask about an obscure one and you might get a convincing blend of three real cases, which is exactly how lawyers have ended up sanctioned for filing briefs with invented citations.

Training rewarded answering. Raters generally preferred confident answers over “I don’t know,” so models drifted toward always having one. Newer models refuse and hedge more appropriately than the 2023 generation, and grounding answers in retrieved documents cuts fabrication a lot. Nothing eliminates it. The practical rule: treat any specific, checkable claim (names, numbers, dates, quotes, citations) as unverified until you’ve confirmed it or the model shows a source.

Context windows: the model’s working memory

The context window is the total amount the model can consider at once: your conversation, pasted documents, its own replies, all counted in tokens. Modern models handle 200,000 tokens and beyond, with some stretching past a million, enough for entire books or codebases.

Two facts about context windows change how you should work. First, the model has no memory beyond them. When ChatGPT seems to remember you across sessions, a memory feature is quietly re-inserting saved notes into context; the underlying network learns nothing from talking to you. Second, big windows aren’t uniformly reliable. Models attend better to the beginning and end of a long context than the middle, so in very long sessions, details from the middle effectively fade. When a long conversation drifts and gets confused, starting fresh with a summary usually beats pushing on.

Reasoning models: teaching LLMs to think longer

The newest generation, OpenAI’s o-series, Claude’s extended thinking, Gemini’s thinking modes, adds a twist: before answering, the model generates internal chains of reasoning, exploring the problem, checking itself, and backtracking, then answers. Labs trained this deliberately, rewarding reasoning steps that reach correct answers on math and code where correctness is checkable.

The plain-English way to see it: standard generation answers at the speed of talking, while reasoning models buy themselves thinking time by writing a scratchpad you mostly don’t see. The gains on hard, multi-step problems are real and large. The costs: slower answers and more compute, which is why chat products route easy questions to fast models and hard ones to reasoning modes. It’s also why the old prompt trick of demanding step-by-step thinking has faded; the model already does it, and better, as we cover in our prompt engineering guide.

Under the hood, it’s still next-token prediction. The tokens being predicted just include the working-out now, and it turns out that predicting your way through the working-out is a very effective way to get the answer right. Whether that counts as reasoning is a philosophy question; the benchmark scores don’t care.

Model sizes, mixture of experts, and open weights

Model names come with size hints (nano, mini, flash, pro) and the trade is exactly what it sounds like: bigger models know more and reason better, smaller ones respond faster and cost a fraction as much. For summarizing an email, the small model is the right tool. For untangling a contract, pay for the big one. Chat products increasingly make this call for you with automatic routing, which is convenient and occasionally the answer to “why did quality suddenly change?”

Two structural developments are worth knowing by name.

Mixture of experts (MoE) is now standard in large models. Instead of one monolithic network where every parameter fires on every token, the model is divided into specialized sub-networks, and a routing layer picks which few of them to consult for each token. The result is a model with a huge total parameter count that only spends a fraction of that compute per token. It’s how modern models got bigger and cheaper to run at the same time, and it’s used across the frontier models from Google, OpenAI, Mistral, and others.

Open-weight models are the other axis. Companies like Meta (Llama), Mistral, and several Chinese labs release their trained parameters for anyone to download and run, in contrast to closed models like GPT and Claude that you can only reach through an API. Open weights mean you can run a capable model on your own hardware, fine-tune it on your own data, and never send a byte to a third party. The strongest closed models still lead on capability, but the gap has narrowed enough that open models handle a large share of real workloads. If that appeals, our guide to running AI models locally covers the practical side.

What LLMs still can’t do

An honest capability map, because knowing the edges matters more than admiring the middle.

They can’t verify. No internal fact-checker exists, and confidence carries no information about accuracy. A model states its wrongest claims in the same assured tone as its rightest ones.

They can’t learn from you. The parameters are frozen after training. Corrections last as long as they stay in context, and no amount of chatting improves the underlying model.

They’re poor at precise computation. Arithmetic on big numbers, exact character counts, strict constraint puzzles: token prediction approximates these rather than computing them. Serious products route such tasks to actual code, and reasoning models partially compensate by writing out the arithmetic longhand.

They inherit their training data’s tilt. Internet text over-represents some perspectives, languages, and demographics, and under-represents others. Labs counteract this with curation and fine-tuning, with partial success. Outputs reflect the distribution of what was written, not the distribution of what’s true or fair.

They don’t have goals, in the ordinary sense. On its own, a model completes text and stops. Wrap it in a loop with tools, though, and it can plan, act, and iterate toward objectives. That wrapper is what people mean by AI agents, and it’s where much of the field’s energy has moved.

What this means for how you use them

Theory should change behavior, so here’s the mapping.

Because models predict rather than retrieve, verify anything specific and consequential: cite-check the citation, run the code, confirm the dosage with an actual reference. Use LLMs as drafters and explainers, not as sources of record.

Because knowledge is frozen, prefer tools with search or retrieval for anything recent, and paste in current source material when accuracy matters. The model reasoning over your document beats the model recalling from compressed memory.

Because everything lives in context, front-load what matters. Give the model the brief, the audience, and the source text rather than hoping it infers them. This is most of what prompt engineering amounts to in practice.

Because sampling is random, regenerate before you judge. A weak first answer is one draw from a distribution, and the second draw is often better. For repeatable workflows, test prompts across multiple runs, as covered in our prompt testing guide.

Because strengths differ by model and mode, match the tool to the task: reasoning modes for hard multi-step problems, fast models for summaries and drafts, search-connected tools for current facts. Our ChatGPT vs Claude comparison breaks down where each shines.

Key takeaways

• LLMs generate text by repeatedly predicting the next token; all capabilities emerge from that single mechanism at massive scale.
• Models read tokens, not words, which explains spelling failures, per-token pricing, and uneven quality across languages.
• Attention, the transformer’s core trick, lets every token consult every other token, capturing context that older architectures lost.
• Training has stages: pretraining builds raw capability, fine-tuning creates the assistant, and human feedback shapes behavior. Knowledge freezes at pretraining.
• Hallucination is the natural output of a plausibility machine at the edge of its knowledge. Verify specifics; trust fluency never.
• Reasoning models improve hard-problem performance by generating hidden working-out before answering. It’s still next-token prediction, aimed well.

Frequently asked questions

What is a large language model in simple terms?

A large language model is a program trained on enormous amounts of text to predict what word fragment comes next. Done at sufficient scale and then tuned to follow instructions, this prediction ability lets it answer questions, write, summarize, and code. ChatGPT, Claude, and Gemini are all built this way.

Do LLMs actually understand what they’re saying?

Contested, and partly a definitional fight. They demonstrably build internal representations of meaning, which is why they resolve ambiguity and track ideas across pages. They also lack grounding in physical experience and can’t verify their own claims. “Understands, in a different and narrower way than humans do” is about as far as the evidence takes anyone.

Why do LLMs make things up?

Because they’re trained to produce plausible text, not verified text. Where training data was thin, the most statistically likely continuation is something that merely sounds right, delivered fluently. Search grounding and citation features reduce this substantially; nothing yet eliminates it.

How is an LLM different from a search engine?

A search engine finds existing documents and shows them to you. An LLM generates new text from patterns compressed into its parameters, with no lookup step and no source attached. The lines have blurred: modern search includes AI summaries, and modern chatbots include search. When provenance matters, prefer the mode that cites.

What does GPT stand for?

Generative Pre-trained Transformer. Generative because it produces text, pre-trained because it first learns from a huge general corpus before any task-specific tuning, and transformer for the neural network architecture underneath.

How much data and money does training take?

Frontier models train on trillions of tokens, the equivalent of tens of millions of books, using thousands of specialized chips for months. Credible public estimates put frontier training runs in the hundreds of millions of dollars, which is why a handful of companies (OpenAI, Google, Anthropic, Meta) dominate and most others build on top of their models or on open-weight alternatives.

Is my data used to train the model when I chat?

It depends on the product and settings. Consumer chat apps may use conversations for training unless you opt out; business and API tiers typically exclude customer data by default. Check the data controls of the specific product, and keep sensitive material out of consumer tiers regardless.

What’s the difference between an LLM and AI?

AI is the umbrella: any technique that gets computers doing tasks associated with intelligence, from chess engines to fraud detection. Machine learning is the subset that learns from data. LLMs are one family within that, specialized in language, currently the most visible family because chat made them usable by everyone.

Conclusion

So how do large language models work? They chop text into tokens, map tokens into a geometry of meaning, use attention to let every piece of text inform every other, and predict what comes next, one token at a time, with knowledge compressed into billions of tuned parameters. Training gives them capability, fine-tuning gives them manners, and nothing gives them a fact-checker, which is why the last step of any serious LLM workflow is still you. None of this diminishes the achievement; prediction at this scale turns out to produce something remarkably capable. It just replaces the two lazy stories, magic intelligence and stupid autocomplete, with the true one, which is stranger than either and much more useful to actually know.