Glossary

A weighting step where each token decides how much to draw from every other token in the context.

One of many parallel attention patterns running at once. Different heads learn different jobs. One tracks which noun a pronoun refers to, another follows grammatical structure, and their results are combined.

The name for this loop: each token the model produces is fed back in as input before it chooses the next one. It writes forward only, and never revises what it already said.

The result of pretraining alone. It continues text rather than answering questions, and will happily extend your prompt with more questions.

Making the model point at the passage it drew an answer from, which is what makes verification cheap enough to actually do.

The writing style is equally assured whether the model is recalling or inventing. Specific-sounding detail is not evidence.

Hard boundaries: length, tone, what to leave out, what to do when information is missing.

The maximum number of tokens the model can consider at once, covering the prompt and the response together.

'Bank' near 'river' and 'bank' near 'deposit' end up as different internal representations, despite being the same token.

One slot in a vector — one of the numbers in the list. A map has two dimensions because a place needs two numbers to pin it down. A word in a real model has hundreds or thousands, each one a different aspect of how the word behaves. Nobody assigns them meanings; they come out of training.

Also used in · Meaning Space

The length of that list, often 768 to 3,072 numbers. Each one is a learned shade of meaning that no human named.

How far apart two vectors are. Close means the model treats the two words as similar; far means it does not. This is the whole reason the map is worth drawing — meaning becomes something you can measure.

A list of numbers representing a piece of text as a point in space. Text with related meaning gets nearby points.

Including one or two input-output examples. The strongest single lever for matching a style or format.

Showing two or three worked examples of the input and the output you want, rather than describing the format in words. Almost always beats an adjective, because 'concise' means something different to everyone.

Continuing training on a narrower set of examples to specialise a finished model. It changes the model's weights permanently, where a prompt changes only that one request.

The act of producing output, one fragment at a time. Nothing is retrieved and nothing is reasoned through in the human sense. It is repeated prediction, which turns out to be enough for a surprising amount of useful work.

Always taking the single highest-probability token. Safe and repetitive, and prone to getting stuck in loops.

Supplying the source material in the prompt so the answer is drawn from your documents rather than from vague recall.

A fluent, confident statement that is not true. It emerges from optimising for plausibility rather than for truth.

Running a trained model to get an answer, as opposed to training it. Every message you send is inference: the weights do not change, they are just used.

Also used in · How It Learned

Further training on example instruction-and-response pairs, which teaches the model to treat input as a request.

Treating the first output as a draft and correcting one specific thing at a time, rather than rewriting from scratch.

The date its training text ends. Anything later has to arrive through your prompt, a search tool, or an attached document.

A system trained on enormous amounts of text to predict what text comes next. Answering, summarising, translating and coding are all that one operation applied repeatedly.

The raw scores the model produces for every token in the vocabulary, before they are turned into probabilities. Each one is a single number saying how well that candidate fits, before any of them become percentages.

A link between words far apart, such as a pronoun and the noun it refers to, or an instruction at the top of a long document.

A measured weakness: facts placed at the very start or very end of a long context get used more reliably than facts buried in the middle. Putting the important instruction last is not superstition.

Two different things share this word. The context window is what the model can see in one request and it vanishes afterwards. A product feature called Memory is a store the product keeps and quietly pastes into your next prompt. The model has no memory of its own either way.

Also used in · Context Window

Summarising older turns, or saving facts to a store and re-injecting them, so key details survive past the window.

The model is the engine. ChatGPT, Claude and Copilot are products built around one: a chat interface, a system prompt, file handling, sometimes web search. Two products can use the same model and behave very differently.

The model's only job: score how likely each possible token is to come next, given everything before it.

The model does not draft the ending first. A confident opening sentence can commit it to a claim it then has to keep supporting.

Stating the exact shape you want: bullets, table columns, word count or JSON keys, so you can use the result directly.

The billions of adjustable numbers inside the model, set during training. They hold no facts in a filing cabinet. They are weights that shape which words become likely, which is why recall is fuzzy.

Humans rank candidate answers, a reward model learns those preferences, and the model is tuned toward them. Shapes tone, format and refusals.

Learning to predict the next token across a vast body of text. Expensive, done once, and the source of general knowledge.

The full set of scores across the vocabulary, adding up to 100%. Usually a handful of tokens hold most of the weight.

Everything sent to the model for one request: your message, the conversation so far, any documents, and instructions the product added invisibly. The model sees one block of text, not a conversation.

Retrieval-Augmented Generation: search your documents first, paste the best passages into the context, then ask the question.

The standard fix for hallucination: search a trusted document store first, paste what you find into the context, then ask the question. The model summarises real text instead of reciting from memory.

Telling the model who it is answering as. It shifts vocabulary and assumptions toward that domain.

Rolling a weighted die over the distribution instead of always taking the favourite. It is why the same prompt gives different answers.

Attention applied within a single passage, so every token attends to every other token, including itself.

Searching by embedding rather than by keyword, so a document matches an idea instead of an exact phrase.

How close two embeddings sit. 'Cancel my subscription' and 'end my plan' land beside each other even though they share almost no words.

The step that turns the model's raw scores into probabilities that add up to 1. Temperature works by scaling the scores before this happens, which is why one dial reshapes the whole distribution.

The model remembers nothing between calls. Continuity is an illusion created by resending the conversation each turn.

Standing instructions placed at the top of the context that set role and rules. It occupies tokens on every single turn.

A dial that sharpens or flattens the probability distribution before a token is picked, usually running from 0 to 2.

Effectively always take the top token. Nearly deterministic, and the right default for extraction, classification and structured output.

A quick estimate for English, where one token works out at roughly 4 characters or 0.75 of a word, which puts 1,000 words of prose at about 1,300 tokens.

The smallest fragment of text a model actually reads. Often a word-piece rather than a word: 'unhelpful' can become un + help + ful.

The integer that stands in for a token. The model works entirely with these numbers; letters are never part of the picture.

The program that chops text into tokens and swaps each one for a number. It is fixed and deterministic, so the same text always produces the same tokens.

A simpler cousin: only ever consider the k highest-scoring tokens, regardless of how the probability is spread.

A second dial that sits alongside temperature. Instead of reshaping all the odds, it throws away the long tail, keeping only the most likely options that together make up p of the probability, then choosing among those.

The architecture built on stacked attention layers. The 'T' in GPT, and the reason models handle long-range references well.

When the transcript no longer fits, the oldest messages are cut. The model cannot tell you what it no longer has.

A list of numbers treated as a single value. A point on a map needs two numbers, east and north, and that pair is a vector. Everything the model knows about a word is stored as one of these lists — just a much longer one.

Also used in · Meaning Space · Attention

The directions between points in the space encode relationships, which is why you can take king, subtract man, add woman, and land next to queen.

A store built to hold embeddings and answer 'what is nearest to this?' quickly. It is the piece that lets a chatbot search your documents by meaning instead of by keyword.

The fixed set of tokens a model can read or write, decided before training and never changed afterwards. Anything you type has to be expressed in these pieces, which is why unusual words get chopped into several.

How much one token is pulled on by another when the model works out what it means. High weight means 'this word matters for reading that one'. Also the general word for the billions of adjustable numbers inside a model, set during training — the context usually makes clear which is meant.

Also used in · How It Learned

To predict text well across billions of examples, a model has to internalise grammar, structure, relationships between ideas and a great deal of world knowledge. Whether that counts as understanding is a live argument; that it is more than a lookup table is not.

Tokenizers are trained mostly on English, so English gets efficient whole-word tokens. The same sentence in Thai or Hindi can cost several times more tokens, and therefore more money and more of the context window.

Counting letters means looking inside a token, which the model cannot do directly. That is why letter-counting and rhyming puzzles trip it up while essays do not.