[feat] Add HunyuanVideo 1.5 T2V training support - #1662
[feat] Add HunyuanVideo 1.5 T2V training support#1662KyleNeverGivesUp wants to merge 10 commits into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
mergify
Bot
added
type: feat
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d0402abd19
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Kyle — pushed a commit to your branch rather than sending a list back. Revert anything you disagree with. The main one: the ByT5 fallback was silent. It's also reachable for a reason unrelated to legacy data. Those columns come from #1663. I merged it locally and re-measured: Also replaced the two And Verified: ruff clean on the changed files, CPU tests 10 passed. Credit where it's due — the subclass boundary is right, overriding only the four methods that actually differ from Wan, and the one shared-file change is a new branch in a dispatch that already carries per-family cases. I checked the forward contract against the inference path rather than taking it on faith: One note, not a request: both configs leave validation commented out, but the templates use Not verified here: the end-to-end overfit (needs #1663's preprocess script), the 8-GPU FSDP config, and |
Pre-commit checks failedHi @KyleNeverGivesUp, the pre-commit checks have failed. To fix them locally: # Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-filesCommon fixes:
After fixing, commit and push the changes. The checks will re-run automatically. For future commits, |
…ents, disable validation, warn on mixed prompt lengths
FineTuneMethod hard-codes conditional=True, so the inherited _requires_negative_conditioning=True loads the ~16GB Qwen encoder on every rank to cache an embedding nothing reads, inflating startup memory past the 48GB the runbook advertises. Opt out in __init__, mirroring ltx2.py, which does the same rather than load Gemma. DMD2 re-enables it from _configure_student_negative_conditioning in its own __init__ when its cfg_uncond policy needs negative prompts.
…h from config Missing ByT5 fields were absorbed into a [B, 0, D] zero-token tensor with no log line. That is indistinguishable from healthy training -- finite grads, falling loss, full speed -- so a run conditioned on nothing looks exactly like a run that works, and the runbook tells the reader a (0, 1472) shape is normal and expected. It is also reachable for a reason that has nothing to do with legacy data: the collator derives its tensor fields from the parquet schema, so if the schema does not declare the text_embedding_2 columns the field is dropped before prepare_batch ever sees it, even when the row carries it. Warn once per model, not once per step; the condition is a property of the dataset, and repeating it every iteration would bury the line it exists to surface. Also replace the two 1472 literals with the config that owns the value. HunyuanVideo15ArchConfig.text_embed_2_dim is what constructs the nn.LayerNorm this tensor has to satisfy, so it is the authority; the encoder-side T5ArchConfig default of 512 named in the old comment is unrelated to it and is not where the width comes from. Verified the config returns 1472, so this is behavior-preserving. And call check_grad_norm_regression in the finetune test. It was imported and never used -- the only one of the six per-method tests that did not call it (ruff F401; test files are excluded from the lint lane, so CI stayed green). It skips cleanly on a device with no seeded reference, so it costs nothing now and guards once one exists.
76bd46a to
a75c8cd
Compare
…ix has landed Both configs disabled validation with the note that TextEncodingStage sized the zero-length ByT5 placeholder from the static T5 default (512) rather than HY1.5's 1472, so a caption without glyph text raised "Given normalized_shape=[1472] ... got [1, 0, 512]". That is fixed: the validation pipeline no longer overwrites the loader-populated text-encoder config. Verified on the rebased branch, on a GB200, both directions: - caption WITH quoted glyph text -> ByT5 (19, 1472), training rc=0, two validation videos rendered through the full pipeline; - caption WITHOUT quoted text -> ByT5 (0, 1472), which is precisely the case the comment says fails -> training rc=0, two videos, no RuntimeError. The second run is the one that matters; the first alone would only have shown that a glyph-bearing caption avoids the placeholder path. Uncomment the block in both configs and correct the runbook's troubleshooting row, which told readers the failure was expected and validation was off for that reason.
|
Thanks, keeping all four. The width one corrects me that the authority is the LayerNorm text_embed_2_dim builds, not the encoder config I was reading. |
|
@SolitaryThinker this has been ready since your commits landed. All |
2 participants
Purpose
Adds HunyuanVideo 1.5 to FastVideo's modular training stack, so teams already training Wan/Cosmos here can fine-tune HY1.5 with the same YAML and preprocessed-parquet workflow, and so HY1.5 becomes eligible for the framework's distillation and RL methods.
Related: #1661 (an inference-side ByT5 sizing bug found while running validation here; not fixed in this PR, see Limitations below).
Changes
Hunyuan15ModelsubclassesWanModeland overrides only what differs:prepare_batch: dual text embeddings (Qwen 3584 plus ByT5 1472) with a zero-token fallback for parquet that predates the ByT5 fields, padding trimmed to the batch's longest valid length, and"hunyuan15"VAE scaling (the parquet stores rawlatent_dist.mode()outputs). Padding is trimmed rather than masked because the HY1.5 DiT has no text mask input and its token refiner mean-pools over every token; withtrain_batch_size: 1this matches inference exactly.predict_noiseand_build_distill_input_kwargs: the HY1.5 forward contract.(B, C, T, H, W)layout with conditioning channels packed to 65 (32 latent, 1 mask, 32 conditioning latent, the last two all zero for T2V, mirroringHy15ImageEncodingStageandDenoisingStage),encoder_hidden_statesas[qwen, byt5], an all-zero SigLIP-shaped placeholder to select the T2V branch, and the discrete 0..1000 timestep.encoder_attention_mask,return_dict, andtimestep_rare deliberately absent, since HY1.5's forward rejects or crashes on them.ensure_negative_conditioning: encodes"."through the same Qwen chat-template path the preprocess script uses. The sharedencode_negative_prompthelper does not fit, because HY1.5's preprocess returns a chat message list, its postprocess takes the attention mask and returns a tuple, and it readshidden_states[-3].init_preprocessors:Qwen2_5_VLArchConfigleavestext_lenat 0, which would size the dataloader's text padding to zero tokens.Supporting changes:
normalize_dit_inputgains a"hunyuan15"branch (latents * scaling_factor).docs/training/hunyuan15_overfit.md.fastvideo/tests/train/methods/grad_norm_regression.pygainsdouble_blocksin_BLOCK_LIST_ATTRS. That list already exists to cover per-family naming; the entry is additive and changes nothing for models usingblocksortransformer_blocks.Test Plan
Test Results
pytest, 12 passed on A100 80GB
The 8 CPU tests check the assembled forward kwargs against
HunyuanVideo15Transformer3DModel.forward's real signature viainspect.signature, so signature drift fails loudly without loading weights. The 4 CUDA tests load the real 8.33B checkpoint: two run a forward pass, two run a full training step and assert block 0's gradients are finite and non-zero.Preprocess, parquet contract
Single-clip overfit, loss
A100 80GB, LoRA rank 64 with AdamW, 180 steps on one clip. Flow-matching loss is noisy step to step because each step samples a random timestep, so these are segment means:
The same code path also runs end to end on aarch64 with Blackwell (DGX Spark, GB10, CUDA 13.0) in full-parameter mode.
Limitations
".". This only matters attraining_cfg_rate > 0, and both configs set it to 0.ensure_negative_conditioningalready implements the inference semantics, so aligning the collate side later is self-contained.Checklist
pre-commit run --all-filesand fixed all issuesFor model/pipeline changes, also check: