Qwen3.5 Architecture Analysis — PR #43830

PR: huggingface/transformers#43830
Author: bozheng-hit (Qwen Team, Alibaba Group)
Status: Open (as of Feb 8, 2026)
Branch: bozheng-hit:qwen3_5 → huggingface:main (17 commits)

Overview

This PR adds two new model families to HuggingFace Transformers:

1. Qwen3.5 (dense) — multimodal VLM with hybrid attention
2. Qwen3.5-MoE — same architecture but with Mixture-of-Experts MLP layers

Both models are vision-language models (VLMs) that combine a text backbone with a vision encoder. The key architectural innovation is a hybrid attention mechanism mixing full softmax attention with Gated DeltaNet linear attention.

Files Changed (28 files)

Inheritance Hierarchy

Qwen3.5 Dense:
  Qwen3_5TextConfig  ← Qwen3NextConfig (inherits, deletes MoE params)
  Qwen3_5VisionConfig ← Qwen3VLVisionConfig (removes deepstack)
  Qwen3_5Config       ← Qwen3VLConfig (composite: text + vision)
  
  Qwen3_5GatedDeltaNet  ← Qwen3NextGatedDeltaNet (refactored projections)
  Qwen3_5Attention      ← Qwen3NextAttention
  Qwen3_5MLP            ← Qwen3NextMLP
  Qwen3_5DecoderLayer   ← GradientCheckpointingLayer (custom, hybrid dispatch)
  Qwen3_5TextModel      ← Qwen3NextModel
  Qwen3_5Model          ← Qwen3VLModel (full VLM)
  Qwen3_5ForCausalLM    ← Qwen3ForCausalLM
  Qwen3_5ForConditionalGeneration ← Qwen3VLForConditionalGeneration

Qwen3.5 MoE:
  Qwen3_5MoeTextConfig  ← Qwen3NextConfig (keeps MoE params)
  Qwen3_5MoeDecoderLayer ← Qwen3NextDecoderLayer (uses SparseMoeBlock)
  Qwen3_5MoeForCausalLM  ← Qwen3NextForCausalLM
  Qwen3_5MoeForConditionalGeneration ← Qwen3VLMoeForConditionalGeneration

Qwen3.5 Dense Text Configuration

Default reference model: Qwen/Qwen3.5-9B-Instruct

Linear Attention Parameters (Gated DeltaNet)

Layer Type Pattern

By default, full_attention_interval=4, producing this repeating pattern across layers:

Layer  0: linear_attention
Layer  1: linear_attention  
Layer  2: linear_attention
Layer  3: full_attention    ← every 4th layer
Layer  4: linear_attention
Layer  5: linear_attention
Layer  6: linear_attention
Layer  7: full_attention    ← every 4th layer
...

Ratio: ~75% linear attention, ~25% full attention (8 full out of 32 layers).

Qwen3.5-MoE Text Configuration

Default reference model: Qwen/Qwen3.5-35B-A3B-Instruct

MoE-Specific Parameters

Naming convention: Qwen3.5-35B-A3B = 35B total params, 3B active per token.

Vision Configuration (shared by both variants)

VLM Composite Config

Key Architectural Features

1. Hybrid Linear + Full Attention

Each decoder layer is either a full_attention layer (standard softmax attention with GQA + RoPE) or a linear_attention layer (Gated DeltaNet). The DecoderLayer.forward() dispatches based on layer_type:

if self.layer_type == "linear_attention":
    hidden_states = self.linear_attn(...)
elif self.layer_type == "full_attention":
    hidden_states, _ = self.self_attn(...)

2. Gated DeltaNet (Linear Attention)

The Qwen3_5GatedDeltaNet module implements a recurrent linear attention mechanism:

• Separate projections: in_proj_qkv (fused Q/K/V), in_proj_z (gate), in_proj_b (beta), in_proj_a (alpha)
• Causal Conv1d on Q/K/V before splitting (kernel size 4)
• Chunk-wise gated delta rule for efficient training; single-step recurrent mode for generation
• Uses A_log and dt_bias learnable parameters (similar to Mamba/S6 style parameterization)
• Key formula: g = -A_log.exp() * softplus(a + dt_bias)

3. Custom DynamicCache

Qwen3_5DynamicCache extends Qwen3NextDynamicCache to handle both:

• Standard KV cache for full attention layers
• Recurrent state (conv_states, recurrent_states) for linear attention layers

4. Multimodal RoPE (M-RoPE)

Position IDs are 3D: (temporal, height, width) — supporting:

• Text-only sequences (identical across all 3 dims)
• Images/videos (spatial 2D + temporal position encoding)
• mrope_section = [11, 11, 10] defines how the 32 RoPE dim pairs are split across the 3 axes
• Only 25% partial rotary factor (64 out of 256 head_dim gets RoPE)

5. RMSNorm with Offset

Qwen3.5 uses (1 + weight) * x normalization (zero-initialized weights → starts as identity), inherited from Qwen3-Next.

6. MoE Architecture (MoE variant)

The MoE variant replaces the standard MLP with SparseMoeBlock:

• 256 routed experts with Top-K=8 routing
• Plus a shared expert always active
• TopKRouter with configurable auxiliary loss
• Fused gate_up_proj (packed) for experts

Model Variants Available

Notable Design Decisions

1. Builds on Qwen3-Next (already merged via PR #40771), which itself is a hybrid attention model — Qwen3.5 refactors the projections and removes MoE params for the dense variant
2. Vision encoder is inherited from Qwen3-VL but removes DeepStack (multi-layer visual feature injection)
3. No intermediate_size in MoE config — replaced by moe_intermediate_size and shared_expert_intermediate_size
4. The dense model deletes MoE-related attributes inherited from Qwen3NextConfig via del self.moe_intermediate_size etc.
5. Uses the modular_ pattern — the source of truth is modular_qwen3_5.py, and modeling_qwen3_5.py is auto-generated