CS 486/686
Pretraining Language Models From First Principles
Lecture 20
How raw text becomes a training signal
Search
›
Uncertainty
›
Decisions
›
Learning
By the end, you can trace one training example
- Turn text into tokens, IDs, and vectors.
- Build next-token targets from one sequence.
- Trace \(\mathbf h_t \rightarrow\) logits \(\rightarrow\) probability \(\rightarrow\) loss.
- Explain how one batch updates every parameter.
- Explain why targets are automatic but data curation is not.
- Contrast parallel training with sequential generation.
L19 built the machine. What teaches it?
tokens
→
causal transformer
→
hidden states
→
training signal?
Give it one task, billions of times: predict what comes next.
A language model predicts a distribution
The robot picked up the cup because it was ___
emptyvery plausible
bluealso plausible
brokenpossible
quantumunlikely
training text continues withempty
The label is the observed next token—not the only meaningful continuation.
Our roadmap: text becomes a gradient
1tokenizetext → IDs
→
2shiftinputs + targets
→
3predictlogits → \(p\)
→
4measure−log p
→
5updategradient step
Tokens are not words
hello≠·helloa leading space matters
CS486→CS | 4 | 8 | 6rare strings split
unbelievable→un | belie | vablepieces need not be morphemes
🤖 你好→tokensbytes let any Unicode text be represented
Token count controls sequence length—and therefore memory and compute.
Learn reusable pieces by merging frequent pairs
start with charactersl | o | w | e | rl | o | w | e | s | t
merge l + olo | w | e | rlo | w | e | s | t
after many mergeslow | erlow | est
Save the learned pieces:the tokenizer can now emit low, er, est, … as individual tokens.
Freeze this piece list, then use the same tokenizer for every language-model example.
Byte-pair encoding intuition; Sennrich, Haddow & Birch, “Neural Machine Translation of Rare Words with Subword Units,” 2016.
A real tokenizer, piece by piecelive
Qwen3 tokenizer · 151,669 entries · · means “this token begins with a space.”
Try on the live slides
CS486·tokenization
Each piece has an integer ID; the same fixed tokenizer is used before every model forward pass.
IDs retrieve learned vectors
tokenIDlookup
The785\(\mathbf E_{785}\)
·robot12305\(\mathbf E_{12305}\)
·empty4287\(\mathbf E_{4287}\)
IDs are addresses.
Nearby numbers need not mean nearby concepts.
Embedding rows are learned.
Pretraining gives the vectors meaning.
embedding vectorswithpositional information→causal transformer
Actual IDs from the Qwen3 tokenizer; vocabulary size 151,669.
Shift once: one sequence gives many labels
text
Therobotpickedupthecup
↓ copy, then shift targets one place left
input \(x\)
Therobotpickedupthe
target \(y\)
robotpickedupthecup
Five observed tokens become five classification labels—no manual annotation.
Teacher forcing predicts all positions together
position
1
2
3
4
5
given token
The
robot
picked
up
the
predict
robot
picked
up
the
cup
may read
1
1–2
1–3
1–4
1–5
One forward pass: all true prefixes are present; the causal mask blocks every future token.
Each hidden state votes over the vocabulary
token IDs\([B,T]\)
→
transformer\(\mathbf H:[B,T,d]\)
→
output head\(\boldsymbol\ell:[B,T,V]\)
\(\boldsymbol\ell_t=\mathbf W_{\text{out}}\mathbf h_t+\mathbf b\)
At every position: \(V\) scores—one for every possible next token.
Logits are unnormalized preferences
toy vocabulary context: The robot picked up the cup because it was
candidatelogitmeaning
empty2.0observed target
blue1.0second choice
broken0.0less preferred
quantum−1.0least preferred
Logits can be negative and need not sum to anything.
Softmax turns those scores into probabilities
\(\displaystyle p_i=\frac{e^{\ell_i}}{\sum_j e^{\ell_j}}\)
\(0.644+0.237+0.087+0.032=1.000\)
The observed token determines the loss
observed next tokenempty
model assigned\(p(\text{empty})=0.644\)
cross-entropy\(-\ln 0.644=0.44\)
\(p(\text{target})=0.90 \Rightarrow L=0.11\)
\(p(\text{target})=0.01 \Rightarrow L=4.61\)
Confident and wrong is expensive.
A text probability is built left to right
\(x_1\)\(x_2\)\(x_3\)…\(x_T\)
\(P(x_1,\ldots,x_T)=P(x_1)\prod_{t=1}^{T-1}P(x_{t+1}\mid x_{\le t})\)
\(P(\text{The})\)×
\(P(\text{robot}\mid\text{The})\)×
\(P(\text{picked}\mid\text{The robot})\)×…
The same conditional probabilities power both training and generation.
Training averages the token losses
\[
\mathcal L(\theta)=-\frac{1}{N}\sum_{i=1}^{N}
\log p_\theta\!\left(y_i\mid\text{context}_i\right)
\]
\(N\)non-padding targets in the batch
lower is betteron held-out validation text
perplexity\(\exp(\mathcal L_{\text{val}})\)
If \(\mathcal L_{\text{val}}=2\), perplexity is \(e^2\approx7.4\): uncertainty like roughly seven equally likely choices.
Inspect a real model’s losslive data
Qwen3-0.6B-Base · click a target token to inspect its prefix, probability, and loss.
picked→up: high \(p\), low loss
The→robot: lower \(p\), higher loss
The live slide compares natural and surprising continuations and reports average NLL and perplexity.
The pretraining loop, line by line
# packed token sequences: one extra token supplies the final targetids = next(token_batches) # [B, T+1]
# every position predicts the token immediately to its rightx, y = ids[:, :-1], ids[:, 1:] # each [B, T]
# one vocabulary-sized score vector at every positionlogits = model(x).logits # [B, T, V]
# average next-token cross-entropy over the whole batchloss = cross_entropy(logits.reshape(-1, V), y.reshape(-1))
# discard gradients left from the previous updateoptimizer.zero_grad()
# autograd sends credit and blame through the entire modelloss.backward()
# update all trainable parameters onceoptimizer.step()
The gradient reaches the whole model
embeddingstoken lookup vectors
←
transformer blocksQ/K/V projections, feed-forward layers, norms
←
output headhidden state → logits
←
lossstart the backward pass here
Important: attention weights are recomputed for each input; the projection matrices that produce them are learned parameters.
Self-supervised does not mean “free”
Targets are automatic
The robot picked→up
No person labels every next token.
Data is not automatic
- permission and licensing
- filtering and deduplication
- privacy, safety, and quality
The objective is cheap to label; the corpus is costly to build well.
Before training: build the token stream
documentslicensed or permitted sources
→
curatefilter + deduplicate
→
tokenizefixed vocabulary
→
packcontext windows
→
batchmany sequences
training tokens
held-out validation tokens
Never evaluate next-token loss on text used for parameter updates.
Scale is a budget: balance model and data
too many parameters
model sizedata
too few tokens to train them well
balanced for fixed compute
model sizetraining tokens
parameters and tokens grow together
rough training compute \(\propto\) parameters \(\times\) training tokens
Hoffmann et al., “Training Compute-Optimal Large Language Models,” 2022.
What can next-token prediction teach?
syntax + styleThe keys are on the table.
associationsWaterloo is in Ontario.
proceduresfor i in range(3): print(i)
To predict well, hidden states must capture many recurring patterns in text and code.
But the objective guarantees only prediction
truthfrequent text can still be false
reasoningpattern completion can fail off-distribution
instructionsa base model is trained to continue, not obey
recencyweights do not know events after training
privacytraining text may be memorized
new contextprivate/current facts must be supplied safely
Low loss is useful—not a certificate of truth or helpfulness.
Pretraining creates a base language model
general textnext-token pretraining
→
base LMcontinues text
→
assistantpost-training teaches interaction
Explain gradient descent:
a base model may continue the document—not necessarily follow the request.
Later: instruction tuning, preference optimization, prompting, RAG, and tools.
Same model, different execution
training
true sequence→all positions→loss→update weights
- true prefixes supplied
- many token predictions in parallel
generation
prompt→one distribution→choose + append↺
- weights fixed
- new tokens arrive sequentially
Choose one token, append, repeatreal model data
Qwen3-0.6B-Base · choose a rule, then click repeatedly to grow the context one token at a time.
To be, or not to→a next-token distribution→be
On the live slide, choose argmax or sampling, then append several successive tokens from a real precomputed trace.
Next: watch Qwen run from the inside
inspecttokenizer, weights, attention
generateappend loop + KV cache
controlgreedy, sampling, temperature, top-k/top-p
conditionchat templates and prompts
L21: Dissecting Qwen3-0.6B.
Exit check: can you trace the signal?
Therobotpicked
robotpickedup
At the final position, the model assigns \(p(\text{up})=0.80\).
targetup
token loss\(-\ln 0.80\approx0.22\)
trainingaverage with other positions, then backpropagate
generationchoose a token, append it, repeat
Text → targets → probabilities → loss → gradients.