Lecture 24
From noise to images to interactive worlds
L23
image + text → answer
L24
text / action / context → image or frame
How can a model turn this sentence into pixels?
many pixel values
many valid images
global and local details
Start from random noise and repeatedly denoise.
Create noisy examples from real images.
\(x_t = \sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon\)
\(x_0\): clean image
\(\epsilon\): Gaussian noise
t: noise level
\(x_t\): noisy image
Drag the noise level and watch a real image dissolve into noise.
Input: noisy image, timestep, optional condition. Output: noise estimate.
\(L = \lVert \epsilon - \hat{\epsilon}_\theta(x_t,t,c)\rVert^2\)
Diffusion training is supervised learning because we know the noise we added.
Apply the denoiser repeatedly from pure noise.
x = randn(shape)
for t in scheduler.timesteps:
eps = denoiser(x, t, condition)
x = scheduler.step(x, eps, t)
image = decode(x)The whole generator is an iterative loop.
A real Stable Diffusion run: noise to image, one step at a time.
slower, often better
faster, distilled behavior
The denoiser predicts direction; the scheduler decides how to move.
Text conditions every denoising step.
The denoiser attends to text embeddings while cleaning the image latent.
what is being cleaned
what it should become
diverse, may ignore prompt
faithful, may distort
\(\hat{\epsilon} = \epsilon_{\text{uncond}} + s(\epsilon_{\text{cond}}-\epsilon_{\text{uncond}})\)
\(\epsilon_{\text{uncond}}\): no prompt
\(\epsilon_{\text{cond}}\): with prompt
\(s\): guidance scale
Amplify the prompt direction.
Same prompt and seed — slide the guidance scale.
pixels
latent
The compressed latent space is where Stable Diffusion works.
Denoise a compressed representation, then decode to an image.
less compute
smaller tensors
higher resolution
prompt embedding
predict noise
step rule
latent to pixels
Prompt: a blue robot holding a cup
Keep known pixels fixed; denoise only the missing or masked region.
rough shape
structure
geometry
body layout
Prompt asks for three cups; image shows two.
Prompts are strong hints, not hard constraints.
Diffusion often struggles with exact words, symbols, fingers, and small objects.
Local detail requires global consistency.
distorted objects
training data patterns
unsafe generation
Video generation adds temporal consistency.
\(P(\text{frame}_t \mid \text{previous frames},\ \text{actions})\)
Like next-token prediction, but outputs are frames or latents.
previous screen + mouse click → next screen
An interactive world model must condition on what the user does.
Instead of manually programming every UI transition, learn to predict the next screen from prior screens and user inputs.
A UI becomes a generative world model.
Real NeuralOS frames: each screen is generated from the last frames and your input.
hidden UI state
mouse/keyboard
next frame
diffusion renderer
state changes incorrectly
mistakes feed back
input misinterpreted
const pipeline = new SDPipeline({
model: "sd-turbo",
provider: "webgpu",
});
const image = await pipeline.generate({ prompt, steps: 1 });Small distilled diffusion models can run locally, but hardware support varies.
text LM inference
VLM understanding
diffusion/world generation
text/code generation
image understanding
image/video generation
interactive prediction
models in action loops
retrieval, tools, safety
score matching / SDEs
safety and deployment
large-scale recipes
working mental model first