Lecture 21
One real Qwen turn, end to end
Same causal architecture; new behavior learned after pretraining.
Same prompt to both models: What is gradient descent?
How does it work? What is the role of the learning rate? How does the learning rate affect the convergence of the algorithm? What is the difference between gradient descent and stochastic gradient descent? …
continues with more question-like text
Gradient descent is a numerical method used in optimization to find the minimum (or maximum) of a function. It is commonly used in machine learning …
answers the question
Real Qwen3-0.6B-Base vs Qwen3-0.6B. Answering is learned from curated (instruction, answer) demonstrations and preference data.
We will inspect the real object at every boundary.
* grouped-query attention — explained in a few slides. Qwen Team, “Qwen3 Technical Report,” 2025.
{
"hidden_size": 1024,
"intermediate_size": 3072,
"num_hidden_layers": 28,
"num_attention_heads": 16,
"num_key_value_heads": 8,
"head_dim": 128,
"vocab_size": 151936,
"tie_word_embeddings": true
}messages = [
{
"role": "user",
"content": "Explain gradient descent in one sentence."
}
]The model still consumes tokens—so this structure must become one string.
<|im_start|>/<|im_end|> are ChatML message-boundary tokens (im = input message); each is one special token in the vocabulary.
No system message appears unless we actually supplied one.
Single turn or multi-turn → Qwen special tokens → thinking ON/OFF generation cue.
<|im_start|>usermessage<|im_end|><|im_start|>assistantThinking OFF pre-fills an empty think block; thinking ON lets Qwen generate the block.
· means a leading space. IDs shown are actual Qwen tokenizer outputs, not illustrative.
The same matrix \(E\) returns at the end to score output tokens.
The abstraction is unchanged; these names specify Qwen’s implementation.
The real model repeats this pattern: 16 query heads grouped over 8 key/value heads.
Every block adds an update; the residual stream carries the evolving \([T,1024]\) representation.
Head A sends it→cup, head B sends it→robot. Switch to the unambiguous control to see what each head really tracks.
On “The farmers loaded the truck because it was empty,” head B still finds the object (truck 0.84) while head A drifts to the start token. A curated head is a measurement, not an explanation.
Use attention to inspect a computation—not to claim the model’s full reason.
Jain & Wallace, “Attention is not Explanation,” NAACL 2019.
Wiegreffe & Pinter, “Attention is not not Explanation,” EMNLP 2019.
Real Qwen trace after the non-thinking template for “Explain gradient descent in one sentence.”
Step through an exact Qwen continuation; every distribution is conditioned on the enlarged context.
Each appended token becomes part of the next model input.
Build hidden states and K/V cache for the whole prompt.
Generate sequentially, one distribution per step.
Parallel prefill; sequential decode.
Every prompt token of Explain gradient descent in one sentence. is available up front.
Prefill = the single forward pass over the known prompt, done in parallel.
Generate word by word: earlier hidden states and K/V are reused from cache, never recomputed.
Because attention is causal, earlier states never change, so decoding reuses them from the cache.
One row per forward pass: with the cache each decode step adds a single new column.
Prefill computes the prompt once; each decode step then adds only one new cache column.
The cache buys speed by spending memory.
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B", device_map="auto")text = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False)→ text[:34] = '<|im_start|>user\nExplain gradient…'inputs = tok(text, return_tensors="pt").to(model.device)→ inputs.input_ids.shape = [1, 20]→ input_ids[0,:6] = [151644, 872, 198, 840, 20772, 20169]all_ids = model.generate(**inputs, max_new_tokens=64)→ all_ids.shape = [1, 48] # 20 prompt + 28 newanswer = tok.decode(all_ids[0, 20:], skip_special_tokens=True)→ answer = "Gradient descent is a method used to minimize a function…"import { pipeline } from "@huggingface/transformers";
const qwen = await pipeline(
"text-generation",
"onnx-community/Qwen3-0.6B-ONNX",
{ device: "webgpu", dtype: "q4f16" },
);
const messages = [{
role: "user",
content: "Explain gradient descent.",
}];
const result = await qwen(messages, {
max_new_tokens: 64,
do_sample: true, temperature: 0.7, top_p: 0.8, top_k: 20,
});
console.log(result[0].generated_text.at(-1).content);The concepts and default demos never depend on the live download succeeding.
Qwen context: Write a creative name for a friendly blue robot.
**The weights and distribution are fixed; only the selection rule changes.
Prompt Explain gradient descent in one sentence. — each preset has its own exact distribution and continuation.
On the live slide, each preset changes the displayed distribution, chosen token, and continuation.
Non-thinking Qwen still lays out the steps over many tokens — step through it or reveal the full generation.
Qwen writes out 48 ÷ 2 = 24, then 48 + 24, and answers 72 clips.
Real greedy Qwen outputs. More instructions and examples are simply more conditioning tokens.
<|im_start|>assistant
<think>
</think>
Template pre-fills an empty <think></think> block, so Qwen goes straight to the answer.
<|im_start|>assistant
Template stops at the cue, so Qwen generates <think>…</think> before the answer.
Reasoning text can help on hard tasks, but costs tokens and is not guaranteed correct or causally faithful.
Ask about the future and watch a confident, invented answer generate token by token — then reveal the full fabricated name.
2031 is in the future; the correct behavior is to abstain or retrieve current evidence.
We steered a fixed model through its inputs and its decoding—nothing more.
L22: when inputs are not enough—prompting vs RAG/tools vs SFT/LoRA, and Qwen as a frozen interpreter for PAW.