From noise to an image
Deng, Kojima & Rush, “Markup-to-Image Diffusion Models with Scheduled Sampling,” ICLR 2023.
Image → video → interactive computer
imagegenerate pixels
→
videoadd time
→
world modeladd actions + feedback
Image generation
Learn a path from noise to data.
Start with noise and an image
Generation learns how to travel from the left endpoint to the right.
Connect them with a pathinteractive
\(z_t=(1-t)z_{\text{noise}}+t z_{\text{data}}\)
Velocity is the slope of the path
\[
z_t=(1-t)z_{\text{noise}}+t z_{\text{data}}
\]
↓ differentiate with respect to \(t\)
\[
v^\star=\frac{dz_t}{dt}=z_{\text{data}}-z_{\text{noise}}
\]
Generation follows those arrows repeatedly
sample noise\(z_0\)
predict velocity\(v_\theta(z_t,t,c)\)
update + repeat\(z_{t+\Delta t}\)
Many small moves turn an easy random sample into a structured output.
Each velocity prediction advances the sample
Video generation
Generate many consistent frames—not just one image.
An image is a one-frame video
image
[C, H, W]→
video
[C, 1, H, W]Add one axis: time.
Video adds time
0
1
2
3
4
same object
smooth motion
consistent scene
Wan2.1-T2V-1.3B
An open text-to-video model built from a video VAE, a DiT, and a text encoder.
Wan’s default clip has 81 frames
frame 0\(t=0\)
80 intervals\(80/16=5\) seconds
frame 80\(t=5\)
81 RGB frames
→
21 latent time steps
Other supported lengths follow \(4k+1\), such as 33, 49, or 81 frames.
The Wan-VAE compresses the video
[3, 81, 480, 832]encoder→
[16, 21, 60, 104]decoder→
[3, 81, 480, 832]
time81 → 21 ≈ ÷4
height480 → 60 ÷8
width832 → 104 ÷8
channels3 → 16 expand
Team Wan et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” 2025; example provenance.
A latent video is a nested array
16 channels
different learned features
21 time steps
inside every channel
60 × 104 matrix
of real numbers
\([16,21,60,104]\quad=\quad z[\text{channel},\text{time},\text{row},\text{column}]\)
Patchify the latent—just like an image encoder
RGB video
[3, 81, 480, 832]
≈ 97 million values
latent grid
[16, 21, 60, 104]
continuous features
patch sequence
[32,760, 64]
\(21\times30\times52\) patches
DiT tokens
[32,760, 1536]
transformer input
One token begins as \(16\) channels × \(1\) time step × \(2\times2\) spatial cells \(=64\) values.
A DiT is a transformer for noisy latent patches
noisy latent patches32,760 tokens
→
Transformerattention + MLP
→
velocity patchessame sequence shape
Peebles & Xie, “Scalable Diffusion Models with Transformers,” ICCV 2023.
Wan2.1-T2V-1.3B at a glance
Wan2.1
T2V-1.3B
Wan-AI/Wan2.1-T2V-1.3B
1.3BDiT parameters
30DiT blocks
1536hidden width
480ptarget resolution
Team Wan et al., “Wan: Open and Advanced Large-Scale Video Generative Models,” 2025; official checkpoint configuration.
Wan uses only the umT5-XXL encoder
umT5-XXL is a ~13B encoder–decoder Transformer; Wan keeps the 5.68B encoder and discards the decoder.
prompt tokens“a blue robot …”
→
5.68B encoder24 Transformer layers
→
contextual states\([T,4096]\)
Chung et al., “UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining,” 2023.
Prompt in; video out
prompt → umT5 hidden states
↓
latent noise\(z_0\)
→
Wan DiTpredict velocity
→
updated latentrepeat until clean ↺
decode once
clean latent\([16,21,60,104]\)
→
Wan-VAEcheap decoder
→
RGB video81 frames
Text conditions the DiT—not the VAE decoder.
A Wan block mixes video, text, and time
video self-attentionmix space + time
text cross-attentionread umT5 states
feed-forward networktransform each token
× 30blocks
The timestep modulates each block, telling it how noisy the current latent is.
Train the DiT to predict velocity
real videoVAE → \(z_{\text{data}}\)
add noisechoose \(t\), build \(z_t\)
Wan DiTpredict \(v_\theta(z_t,t,c)\)
target slope\(z_{\text{data}}-z_{\text{noise}}\)
\[
\mathcal L=\left\lVert v_\theta(z_t,t,c)-(z_{\text{data}}-z_{\text{noise}})\right\rVert_2^2
\]
Train the DiT. Keep umT5 and the Wan-VAE fixed.
The generated resultrecorded model output
Prompt
A small blue robot writes E = mc² on a blackboard in a sunlit university classroom, then turns toward the camera and waves, fixed camera, coherent motion, clean cinematic lighting.
Generated with Wan2.1-T2V-1.3B; reproducibility record.
From video generation to world modeling
A fixed video cannot react. A world model observes actions and generates what happens next.
What if the whole computer were a neural network?
Mouse and keyboard go in. The next screen is generated.
Rivard et al., “NeuralOS: Towards Simulating Operating Systems via Neural Generative Models,” ICLR 2026.
A video generation model is not a world model
video generator
prompt once
fixed clip
world model
observe each action
predict → feed back ↺
A generated video is static. You cannot interact with it like a video game.
NeuralOS predicts the next screen
double-click Home→
neural
network
network
→
\((\text{screen history},\ \text{action})\rightarrow\text{next screen}\)
A new action generates a new frameinteractive
Click an action. The model generates the next screen—there is no autoplay.
current framenew actiondouble-click Home→
generated framenew actionclick ×→
generated frameNext word prediction → next frame prediction
words so far“the robot”
→
next word“waves”
screens so far + actiondesktop + double-click
→
next framefile manager
Wan can be adapted to predict what happens next
Why reuse Wan? Video pretraining has already learned useful priors about appearance, motion, and dynamics.
history + new actionframes, mouse, keyboard
→
Wan backboneadapt the pretrained DiT
→
next frame or chunkgenerated response
NeuralOS uses a different backbone. The shared formulation is repeated next-frame prediction.
Zhao et al., “minWM,” 2026; He et al., “Matrix-Game 2.0,” 2025.