How Large Language Models Actually Work

What an LLM actually does

At its core, a large language model is a prediction engine. Given a sequence of tokens — fragments of words, punctuation, and symbols — it estimates the probability of every possible next token and selects one. Repeat that process thousands of times and you get a paragraph, an email draft, or a code snippet.

This is why models can be eloquent without being truthful. They optimise for plausible continuation, not verified fact. Understanding that distinction is the foundation for using LLMs responsibly in school, at work, and in product design.

• Training teaches pattern recognition across massive text corpora — not a database of facts.
• Inference is autoregressive: each generated token becomes input for the next prediction.
• The model has no persistent memory between sessions unless you or the application provide context.

From text to tokens

Before a model sees your prompt, a tokenizer breaks text into tokens. Common words may be one token; rare words split into several. Token limits (context windows) are measured in tokens, not words — which is why long documents or code files can hit limits faster than expected.

Tokenization affects cost, speed, and behaviour. Two prompts that look similar to humans can tokenize differently and produce different results. When debugging odd outputs, checking token count and prompt structure is often more useful than rewriting the entire request.

• Rough rule: 1 token ≈ 0.75 English words, but varies by language and formatting.
• Code, JSON, and URLs often consume more tokens per visible character than prose.
• System prompts, chat history, and retrieved documents all share the same context budget.

Attention: how context gets weighted

Transformer architectures use attention mechanisms to decide which parts of the input matter most when predicting the next token. When you ask a model to "summarise the third paragraph," attention is how earlier tokens influence the prediction — not keyword search, but learned relevance across the whole sequence.

You do not need matrix algebra to use this insight. Practically: put critical instructions and constraints near the beginning or end of the prompt, keep unrelated text out of context, and structure long inputs with clear headings so the model can locate the right section.

• Attention scales with context length — very long inputs can dilute focus on key details.
• Multi-turn chats accumulate context; old turns can steer answers in unintended directions.
• Retrieval-augmented setups prepend external chunks — attention then weighs those alongside your question.

Training vs inference

Training is the expensive phase: models adjust billions of parameters by predicting masked or next tokens across internet-scale datasets. Inference is the cheap phase you use daily: a frozen model runs forward passes on your prompt. Fine-tuning and RLHF sit between these extremes, nudging behaviour without retraining from scratch.

For most teams, the actionable split is simple: you rarely train foundation models yourself; you choose a model, craft prompts, add retrieval or tools, and evaluate outputs against real tasks.

• Base models predict text; instruction-tuned models follow conversational formats more reliably.
• Temperature and top-p sampling control randomness — lower values for factual tasks, higher for brainstorming.
• Latency and cost grow with model size, context length, and output length — not just "smarter = better."

A practical mental model for daily use

Treat the LLM as a skilled improv partner with encyclopaedic exposure to language patterns but no guaranteed access to current facts or your private data. Your job is to supply context, specify format, define success criteria, and verify outputs — especially for high-stakes decisions.

When outputs fail, walk the pipeline backwards: Was the prompt ambiguous? Was context missing? Was the task beyond the model's reliable capability? That debugging habit separates productive AI use from frustrated trial-and-error.