Skip to content

Add Qwen3-30A3B AttnRes research experiment - #41

Open
Randomizez wants to merge 1 commit into
devfrom
zhouhy/attn-res-qwen3-30a3b
Open

Add Qwen3-30A3B AttnRes research experiment#41
Randomizez wants to merge 1 commit into
devfrom
zhouhy/attn-res-qwen3-30a3b

Conversation

@Randomizez

@Randomizez Randomizez commented Mar 18, 2026

Copy link
Copy Markdown
Collaborator

What

This PR adds a research experiment for Attention Residuals on Qwen3-30A3B:

  • new exp: playground/research/attn_res/qwen3_30a3b_attn_res.py
  • supports:
    • attn_res_mode = "none" | "full" | "block"
    • block-mode residual routing with configurable attn_res_num_blocks
  • keeps the existing Qwen HF reshaper path by subclassing QwenModel
  • uses the existing Qwen3-30A3B SFT setup as the experiment base

Implementation notes

  • AttnRes is applied at sub-layer granularity
    • one routing op before attention
    • one routing op before FFN
  • effective AttnRes depth is therefore 2 * num_layers
  • current implementation is restricted to:
    • PP=1
    • VPP=1
  • for TP sequence parallel, the new pseudo_query parameter is marked so its gradients are synchronized across the TP group

Docs

The new experiment file starts with a short module doc that summarizes:

  • the paper-style routing equations
  • Full AttnRes pseudocode
  • Block AttnRes pseudocode

Validation

Local checks performed:

  • import/config smoke check
  • cfshow config inspection
  • scratch-init multi-GPU smoke runs using repo .venv/bin/torchrun

Scratch smoke runs after init cleanup were finite:

  • seq_len=256: finite loss / grad norm
  • seq_len=1024: finite loss / grad norm

Known limitations

  • no real base checkpoint path is wired yet in this environment for Qwen3-30B-A3B-Base
  • current PP support is intentionally disabled because AttnRes source-history semantics across pipeline stages are not implemented yet

Extra

This PR also records a few repo priors in AGENTS.md that came up while implementing and validating this experiment.

@Randomizez
Randomizez requested a review from a team March 18, 2026 06:29
@Randomizez
Randomizez force-pushed the zhouhy/attn-res-qwen3-30a3b branch from e55afbd to c491ff2 Compare March 18, 2026 06:34
@Randomizez Randomizez changed the title exp: attn res qwen3 30a3b Add Qwen3-30A3B AttnRes research experiment Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants