Without an image, Qwen is the LM you already know

L21 already explained this path. Today we change the input, not the output loop.

The exact model we will dissect

Qwen Team, “Qwen3.5: Towards Native Multimodal Agents,” 2026; official model configuration.

One twist: most blocks carry a recurrent state

Long image contexts do not pay full quadratic attention in every block.

Make the image look like language to the LLM

Pixels and token IDs begin in different worlds

First compress local pixels into a sequence of patch vectors.

Qwen starts with 16×16 image patches

one patch\(16\times16\times3=768\) pixel values

one learned vectorthe patch embedding

all patchesa spatial sequence

A fixed square resize can erase small details

→

Documents need enough retained patches for their text to survive.

“Native resolution” means a variable patch grid

The processor does not promise to retain every original pixel; it chooses the largest useful grid within its configured budget.

SigLIP 2 NaFlex and Qwen multimodal image preprocessing use variable grids to reduce aspect-ratio distortion.

More retained pixels create a longer visual sequence

Resolution buys detail by spending context and compute.

Qwen merges each 2×2 neighborhood

Recall CLIP: put matching images and text nearby

image encoder \(f_I(I)\) → blue point

image text similar vectors

“a person wearing a suit”

text encoder purple point ← \(f_T(T)\)

Radford et al., “Learning Transferable Visual Models From Natural Language Supervision,” 2021. Callback to L18.

Contrastive training turns appearance into meaning

Now the vision features are useful to a language model—not merely compact pixels.

Qwen’s vision tower is derived from SigLIP2

We use the idea and move on; the SigLIP2 training refinements are optional reading.

Tschannen et al., “SigLIP 2,” 2025; Qwen3.5 vision configuration.

The vision tower contextualizes every patch

The same tower also handles video with temporal patches; today we follow one still image.

The connector changes count and width

Qwen3.5 visual merger: LayerNorm, 2×2 spatial merge, two linear layers with GELU.

Same shape means the same LLM interface

llm(inputs_embeds = concat(image_embeds, text_embeds))

To the backbone, both are sequences of 1024-dimensional vectors.

The image and question become one context

Image spans can appear wherever the prompt places them. After embedding, every position uses the same 1024-D interface.

Once embedded, generation is unchanged

Vision changes prefill; decoding still grows the text answer one token at a time.

End-to-end training teaches Qwen to use the adapter

Where are Wednesday’s office hours?

They are in DC 2633.

Early-fusion multimodal training makes the connected system useful—not the connector shape alone.

One image, every boundary

The recorded trace is generated from the real processor; the demo reports its actual grid for uploaded images.

The complete browser path is short

const processor = await AutoProcessor.from_pretrained(MODEL);
const model = await Qwen3_5ForConditionalGeneration
  .from_pretrained(MODEL, {
    device: "webgpu",
    dtype: {
      embed_tokens: "q4",
      vision_encoder: "fp16",
      decoder_model_merged: "q4",
    },
  });

const prompt = processor.apply_chat_template(messages, {
  add_generation_prompt: true,
  enable_thinking: false,
});
const inputs = await processor(prompt, [image]);
const output = await model.generate({ ...inputs, max_new_tokens: 64 });

Adapted from the official Transformers.js Qwen3.5 WebGPU example.

Too few patches can erase the evidence

The model cannot recover visual evidence that preprocessing removed.