Skip to content

[new-model] Added LingBot-World-Fast image-to-video support - #1665

Open
Ishxn20 wants to merge 2 commits into
hao-ai-lab:mainfrom
Ishxn20:add-lingbotworld-fast-model
Open

[new-model] Added LingBot-World-Fast image-to-video support#1665
Ishxn20 wants to merge 2 commits into
hao-ai-lab:mainfrom
Ishxn20:add-lingbotworld-fast-model

Conversation

@Ishxn20

@Ishxn20 Ishxn20 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Wires up FastVideo/LingBot-World-Fast-Diffusers, the old LingBot-World repo's distilled causal I2V checkpoint (generate_fast.py / LingBot-World-Fast) that was never ported.
  • Reuses the existing LingBot World 2 causal-fast DiT unchanged — verified all 1421 parameter names and shapes match the released checkpoint exactly. Only the arch config differs: local_attn_size=-1 (global attention, no eviction), chunk_size=3, timesteps_index=(0,179,358,679).
  • Adds the arch/pipeline configs, registry entries, preset, pipeline subclass, example script, and SSIM test needed to run it.
  • Fixes a model-family detector collision: the LingBot World (Cam) detector only excluded the "causal-fast" spelling, so it would have wrongly claimed this checkpoint's path and its lingbotworldcausaldmdpipeline model_index name. Registered LingBot-World-Fast first and tightened the exclusion.
  • Fixes WanCrossAttention in the shared LingBot World 2 causal-fast DiT to route through the attention() dispatcher instead of calling flash_attention() directly, so the already-advertised TORCH_SDPA backend actually works when flash-attn isn't installed. Self-attention already used this dispatcher; cross-attention did not.

Test plan

  • Verified all 1421 transformer parameter names and shapes against the real downloaded checkpoint (name + shape match, exact)
  • Verified effective config after update_model_arch matches the reference generate_fast.py (chunk_size=3, timesteps_index=(0,179,358,679), sample_shift=10.0)
  • Verified registry resolution for all three LingBot checkpoints (Base-Cam, Fast, v2 causal-fast) — no cross-contamination
  • pre-commit run --files ... clean (yapf, ruff, codespell, mypy)
  • Loaded real weights end-to-end (18.54B params, 16 shards) and generated a short video on a single GPU (no FSDP, no flash-attn — SDPA fallback) — produced a coherent, prompt-following video with correct camera motion
  • SSIM regression test — needs 2 GPUs, unavailable in dev environment; will run in CI. First reference video should upload as a draft since this is tagged [new-model] (bootstrap mode)
  • Activation-trace parity against the original wan/modules/model_fast.py — the original also requires flash-attn, unavailable in dev environment

Wired up FastVideo/LingBot-World-Fast-Diffusers, the old LingBot-World
repo's distilled causal I2V checkpoint that was never ported. Reused
the existing LingBot World 2 causal-fast DiT unchanged (verified all
1421 parameter names and shapes against the released checkpoint) and
added the arch/pipeline configs, registry entries, preset, pipeline
subclass, example, and SSIM test needed to run it.

Fixed a model-family detector collision where the LingBot World (Cam)
detector would have wrongly claimed this checkpoint's path and
model_index name, since it only excluded the "causal-fast" spelling.

Fixed WanCrossAttention in the shared LingBot World 2 causal-fast DiT
to route through the attention() dispatcher instead of calling
flash_attention() directly, so the already-advertised TORCH_SDPA
backend actually works when flash-attn isn't installed. Self-attention
already used this dispatcher; cross-attention did not.

Verified end to end: weights load (18.54B params, 16 shards) and
generation produces a coherent video with correct prompt-following
camera motion. Not yet verified against the original implementation's
numerics (activation-trace parity) or via the SSIM regression test,
both of which need hardware unavailable in this environment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 30, 2026 01:23
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@mergify mergify Bot added type: new-model New model support scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) labels Jul 30, 2026
@mergify

mergify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

Protection Waiting on
🔴 PR merge requirements 👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

  • #approved-reviews-by>=1
  • check-success=full-suite-passed
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=full-suite-passed
  • check-success=fastcheck-passed
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

@mergify

mergify Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Pre-commit checks failed

Hi @Ishxn20, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

  • yapf: yapf -i <file> (formatting)
  • ruff: ruff check --fix <file> (linting)
  • codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

Ran pre-commit on the full PR diff after the fact and caught a line
yapf wanted collapsed that a later edit had left split across two
lines. No functional change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new LingBot-World-Fast image-to-video model variant to FastVideo by introducing a dedicated pipeline/config/preset/registry wiring, plus an SSIM similarity test and a small attention-path fix to make the shared causal-fast DiT work without FlashAttention installed.

Changes:

  • Registers the new FastVideo/LingBot-World-Fast-Diffusers variant (configs + preset + pipeline) and adjusts LingBotWorld detector ordering/exclusions to avoid resolution collisions.
  • Adds a LingBot-World-Fast pipeline subclass that forcibly swaps in FlowUniPCMultistepScheduler to match the reference sampling path.
  • Updates LingBotWorld2 causal-fast cross-attention to route through the local attention() dispatcher (FlashAttention if available, SDPA fallback otherwise) and adds an SSIM regression test for the new model.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
fastvideo/tests/ssim/test_lingbot_fast_similarity.py New SSIM similarity test for LingBot-World-Fast I2V.
fastvideo/registry.py Registers the new model family first and tightens LingBotWorld base detector exclusions; wires presets.
fastvideo/pipelines/basic/lingbotworld_fast/presets.py Adds lingbotworld_fast_i2v inference preset for the new family.
fastvideo/pipelines/basic/lingbotworld_fast/fast_pipeline.py New pipeline subclass that replaces the scheduler with FlowUniPCMultistepScheduler.
fastvideo/pipelines/basic/lingbotworld_fast/init.py Exports/declares EntryClass for the new pipeline package.
fastvideo/models/registry.py Adds a model-class alias mapping for the checkpoint’s shipped DiT class name.
fastvideo/models/dits/lingbotworld2/causal_fast.py Routes cross-attention through attention() dispatcher (enabling SDPA fallback).
fastvideo/configs/pipelines/lingbotworld_fast.py New pipeline config specializing LingBotWorld2 causal-fast config for LingBot-World-Fast.
fastvideo/configs/pipelines/init.py Exposes the new pipeline config in the configs package exports.
fastvideo/configs/models/dits/lingbotworld_fast.py Adds a LingBot-World-Fast DiT arch config with updated attention/sampling parameters.
fastvideo/configs/models/dits/init.py Exposes the new DiT config in the DiT configs package exports.
examples/inference/basic/basic_lingbotworld_fast.py Adds a basic end-to-end inference example script for the new model.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +28 to +33
logger = init_logger(__name__)

REQUIRED_GPUS = 2

# The released checkpoint is 4-step distilled; see LingBotWorldFastArchConfig.
NUM_DISTILLED_STEPS = 4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) type: new-model New model support

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants