Tokens

Before a language model can process text, it must convert that text into a form it can do math on. Text as you type it is a string of characters. The model needs numbers. The first step is : breaking input text into small chunks called tokens and mapping each to a unique integer.

How Byte Pair Encoding Works

The method behind most modern tokenizers is called (BPE). The core idea is elegant:

1. Start with the raw characters of a language
2. Scan a large text corpus for the two characters that appear side by side most frequently
3. Merge that pair into a single new token
4. Repeat: find the next most frequent adjacent pair, merge it
5. Continue for tens of thousands of iterations until you reach the desired vocabulary size

The result is a vocabulary that mirrors the actual patterns of the language. Extremely common sequences like "ing", "the", or "tion" become single tokens because they appeared constantly during merging. Rare sequences stay split into smaller pieces because they never triggered a merge.

Vocabulary sizes have grown dramatically - an eightfold increase from Llama 2 to Gemini in just two years. Larger vocabularies mean shorter token sequences for the same text, which reduces computation in the .

The Practical Token Math

In clean English prose, one token is roughly three-quarters of a word. So 1,000 tokens ≈ 750 words. But this ratio shifts dramatically with different content types.

Tokenization Pitfalls

Tokenization is not just a preprocessing step. It creates concrete failure modes that affect accuracy, cost, and quality. Understanding these pitfalls is essential for anyone building with .

The Number Problem

Numbers are where tokenization fails most visibly. The token "480" might be a single token, while "481" splits into "4" and "81". The model never sees digits aligned for column addition.

Research at ICLR 2025 demonstrated that simply reversing the tokenization order (right-to-left instead of left-to-right) improves arithmetic accuracy by over 22 percentage points. The math is the same. The are different. The answers change.

Code Is Expensive

Python code tokenizes far less efficiently than English prose. Variable names, operators, brackets, and indentation each become separate tokens. A single line of code can easily consume 15-20 tokens.

The Multilingual Gap

Most vocabularies were built primarily on English text. When you send text in an underrepresented language, individual words fragment into many small tokens - each covering just a few characters. This means:

• More tokens → more computation per request
• Higher API costs for the same semantic content
• Lower quality output because the model has less room in its

Embeddings

After , the model has a sequence of integers. But integer IDs carry no semantic information - the number 472 being close to 473 says nothing about their meanings. The tokens for "happy" and "joyful" might have IDs thousands apart, despite meaning nearly the same thing.

Before any processing begins, each integer token gets converted into a : a list of floating-point numbers, typically between 768 and 12,288 values depending on model size.

Meaning as Geometry

The intuition: imagine placing every word in the English language as a point in a high-dimensional space. Similar words cluster together. Unrelated words sit far apart. But words can be similar in many ways simultaneously - meaning, grammar, category, analogy - and with enough , you can capture all of these relationships at once.

This famous result - king − man + woman ≈ queen - demonstrated in 2013 that vector arithmetic on embeddings produces semantically meaningful results. There exists a direction in the space corresponding to gender, and arithmetic along that direction transforms concepts accordingly.

Static vs. Contextual Embeddings

The king/queen example uses static embeddings, where each word has one fixed vector. Modern language models are more sophisticated - the same token gets different vector representations depending on context.

The process that updates these representations - transforming static token vectors into rich, context-aware representations - is the .

Self-Attention

The 2017 paper that introduced the architecture was titled "Attention Is All You Need." It was not modest. It was also correct.

The core problem attention solves: when processing a given token, which other tokens in the sequence are most relevant to understanding it?

The Coreference Problem

Consider this sentence: "The animal didn't cross the street because it was too tired."

What does "it" refer to? The animal, not the street. You resolved that immediately. A model has to do the same thing computationally. is the mechanism that makes this possible.

Query, Key, Value

For every token, the model computes three vectors from its :

• Query (Q): What this token is looking for
• Key (K): What this token is advertising about itself
• Value (V): What this token actually contributes if attended to

The computation happens simultaneously for every token attending to every other token - all at once, not sequentially. This is what makes transformers parallelizable and fast.

Why Attention Replaced Everything Before It

processed tokens sequentially. To get information from the beginning of a long sequence to the end, it had to pass through every intermediate step - and in practice, it faded. Early context degraded or disappeared entirely.

Self-attention eliminates this distance problem. Every token can directly attend to every other token in a single operation, regardless of distance.

Multi-Head Attention

A single computation can only capture one type of relationship at a time. But language has many simultaneous relationships: syntax, semantics, coreference, temporal ordering, and more. The solution: run many attention heads in parallel.

How It Works

Large models run 32 to 96 in parallel. Each head has its own set of weight matrices, so each head learns to attend to different types of patterns.

Heads Specialize for Specific Tasks

research in 2025, studying individual attention heads in Mistral-7B, found that specific heads specialize for concrete semantic categories:

The Concatenation Step

After all heads compute independently, their outputs are concatenated back into a single vector of the original model dimension, then passed through a final linear projection. This lets the model combine insights from all heads into a unified representation.

Transformer Architecture

is the core mechanism, but a complete block pairs it with several other components that make deep networks trainable.

The Four Components

One transformer block consists of:

1. Multi-Head Self-Attention - each token gathers context from all others
2. Feedforward Network (FFN) - two linear layers with a nonlinearity, applied independently to each token
3. Residual Connections - shortcut paths that add the input directly to the output
4. Layer Normalization - keeps activation values in a stable range

Why Residual Connections Matter

Without , training a 100+ layer network would be impossible. Gradients would vanish as they propagated backward through dozens of layers. The residual shortcut gives gradients a direct highway back to earlier layers.

Why the FFN Is Bigger Than You'd Expect

The in each block expands to 4× the model dimension, then projects back down. In a model with d_model = 4096, the FFN's hidden layer is 16,384 dimensions. This expansion is where much of the model's factual knowledge is believed to be stored - as patterns in the weight matrices.

Depth & the Residual Stream

Large models stack 80 to 120 . What is actually happening across all those layers?

What Each Layer Knows

research has established a consistent pattern:

Visualizing the Stream

Practical Implications

This cumulative architecture has important consequences:

1. No single "understanding" layer exists. Comprehension builds gradually - you can't point to one layer and say "this is where the model understands the sentence."

2. Early exit is possible. For simple tasks, the model may have enough information after 30 layers. Research on "early exit" strategies shows you can skip later layers on easy inputs to save computation.

3. Layer pruning works selectively. Some layers contribute more than others. Studies show you can remove certain middle layers with minimal quality loss - they were adding redundant information to the stream.

Training & Scaling Laws

The training objective for language models is almost insultingly simple: predict the next .

The Training Loop

Take an enormous corpus of text - hundreds of billions to trillions of words. Feed sequences of tokens into the model. At each position, ask: what does the model predict comes next? Compare the prediction to the actual next token. Compute the error. Propagate that error backward through all layers. Adjust each weight by a tiny amount in the direction that reduces the error.

Repeat across enough token positions and the model's weights organize into a compressed representation of enormous amounts of human knowledge.

Chinchilla Scaling Laws

In 2022, DeepMind published the : the compute-optimal training ratio is approximately 20 tokens per parameter. A 70B parameter model should train on ~1.4 trillion tokens.

Pre-training vs. Post-training

on next-token prediction teaches the model what language looks like. After that, most models go through , where human preferences guide fine-tuning toward useful behavior.

Pre-training establishes capability. Post-training shapes behavior.

Generation & Sampling

When you submit a prompt, here is what happens:

1. Your input gets and
2. The sequence passes through all
3. The final block outputs a probability distribution over the vocabulary (32K-262K entries)
4. The model samples one token from that distribution
5. That token gets appended to the sequence
6. Steps 2-5 repeat until an end-of-sequence token or length limit

This is . One token at a time, each feeding into the next pass.

Temperature Controls Randomness

divides the (raw scores) before the function:

Top-p (Nucleus) Sampling

Most production systems combine temperature with :

Emergent Reasoning

Is a language model actually reasoning? Or is it a very sophisticated pattern matcher producing text that resembles reasoning?

The question has gotten considerably more interesting in the past year.

Chain-of-Thought Prompting

When you ask a model to think step by step, performance on complex tasks improves dramatically. This is not just stylistic - writing out intermediate steps forces the model to generate that encode intermediate results, which become context for the next steps.

DeepSeek-R1: Reasoning from Reinforcement Learning

The chain-of-thought research assumed that teaching reasoning required human-labeled reasoning examples. DeepSeek's R1 paper in January 2025 challenged this assumption.

What emerged was striking. The model began generating longer responses that incorporated:

• Self-verification - checking its own intermediate results
• Self-reflection - recognizing when an approach isn't working
• Dynamic strategy adaptation - switching methods mid-solution

The researchers tracked the word "wait" (appearing as a self-correction signal) across training steps:

The model discovered, without being told, that pausing to reconsider was useful. The 77.9% score on AIME 2024 exceeds the average performance of human competitors.

What This Means

The behaviors researchers assumed required human demonstration emerged from a reward signal alone. The model was not told to reflect. It was told to get the right answer. Reflection was what it discovered.

Interpretability

Emergent reasoning is striking. But what is actually happening inside the model while it produces these outputs?

The Polysemanticity Problem

Individual neurons in a language model are : they activate for many unrelated concepts at once. A single neuron might fire for both "legal documents" and "cooking recipes" and "the color blue." This makes direct interpretation of individual neurons nearly impossible.

What Sparse Autoencoders Found

Anthropic's work applying to Claude 3 Sonnet found features corresponding to surprisingly abstract, high-level concepts:

The Honest Position

The distinction between reasoning and very sophisticated pattern matching may be less clean than it sounds. The question of where pattern-matching ends and genuine reasoning begins has not been settled for biological systems either.

What is practically useful:

1. Reasoning quality is sensitive to context. The you provide serve as working memory. Structure what the model sees and you structure what it produces.

2. Internal representations are richer than expected. The model has abstract features, not just surface patterns.

3. The debate may be the wrong question. What matters for practitioners: does the model produce correct, useful output for your task? Understanding the mechanisms helps you engineer better inputs and evaluate outputs more accurately.