MiniMax H3 T2VA, FL2VA, and Ref2VA DiffusionNFT

Last updated: 09/23/2026

These recipes train rank-64 MiniMax H3 LoRA adapters with online DiffusionNFT for text-to-audio-video (T2VA), first-frame image-to-audio-video (FL2VA), and reference-to-audio-video (Ref2VA) generation. A Diffusers transformer is trained with FSDP2 while vLLM-Omni generates joint video and audio rollouts. CLAP and ImageBind provide the default multi-reward (audio-video alignment).

A ready-made FL2VA first-frame dataset built with the data pipeline below is published at https://huggingface.co/datasets/zyfenghit/dancegrpo-t2av

Video Examples

ID	Prompt	MiniMax H3 base model	MiniMax H3 + DiffusionNFT
1	stickman monigote shooting a energy sphere from his hands
2	a husky dog with sunglasses riding on santas sled
3	minimalist polygonal human skull in green flames with strong movement, uhd
4	17th century sailing ship making a path through the waves during a storm
5	close up of a skin texture, two hands with black gloves tatoo a red butterfly over it

Install

Follow the project installation guide.

Checkpoint

MODEL_PATH must be a local MiniMax-H3 repo root containing FL2VA/ (vLLM-Omni rollout checkpoint) and transformer/ (converted Diffusers MiniMaxH3Transformer3DModel for FSDP training). Do not replace the official rollout transformer with a symlink to the Diffusers conversion.

Data Preparation

T2VA (prompt-only)

Convert prompt splits to prompt-only parquet (no condition images, and no negative prompts, since H3 is CFG-distilled):

python3 examples/diffusionnft_trainer/minimax_h3/prepare_t2va_data.py \
    --input_dir /path/to/raw_prompts \
    --output_dir /path/to/h3_t2va_data

Input is train.txt/test.txt (one prompt per line) or train.jsonl/test.jsonl (prompt/text/caption fields).

FL2VA first-frame data

Offline pipeline for MiniMax H3 FL2VA (text+image to audio-video) RL training: turn a prompt list into FLUX reference images, pair them into train/test JSONL, and feed the FL2VA prepare_data.py converter.

Pipeline

prompts.txt ──► gen_flux_images.py ──► images/{index:06d}.jpg
                     │                        │
                     │                        ▼
                     │              build_fl2va_jsonl.py
                     │                        │
                     ▼                        ▼
              (same index)          train.jsonl / test.jsonl
                                             │
                                             ▼
                              prepare_data.py --frame_mode first
                                             │
                                             ▼
                                train.parquet / test.parquet

Getting the prompt file

dancegrpo_consist-id.txt is the filtered ConsisID prompt list released by DanceGRPO (27,815 prompts, one per line). Download it directly from the DanceGRPO repository:

curl -L -o dancegrpo_consist-id.txt \
  https://raw.githubusercontent.com/XueZeyue/DanceGRPO/main/assets/consist-id.txt

The prompts originate from ConsisID-preview-Data captions; DanceGRPO ships the filtered result as-is and does not document the exact filtering criteria, so downloading the released file is the reproducible way to get the identical prompt set (verified line-for-line against the copy used for the published dataset).

Scripts

`gen_flux_images.py`

Multi-GPU FLUX batch image generator. Reads one prompt per line, shards the prompts across ranks (torchrun), and writes one JPEG per prompt plus a per-rank metadata_rankN.jsonl (image <-> prompt <-> index mapping). Deterministic per-prompt seed (seed + index), and re-running skips images that already exist, so interrupted jobs resume safely. Defaults mirror DanceGRPO’s online reference pipeline (400x640, 30 steps, guidance 3.5, max_sequence_length 512), so prompt and condition image are semantically aligned by construction.

torchrun --nproc_per_node=8 examples/diffusionnft_trainer/minimax_h3/gen_flux_images.py \
    --prompt_file dancegrpo_consist-id.txt  # see "Getting the prompt file" \
    --model_path /path/to/FLUX.1-dev \
    --output_dir data/flux_images \
    --height 400 --width 640

`build_fl2va_jsonl.py`

Pairs each prompt with its same-index image, verifies all images exist, shuffles with a fixed seed, and writes train.jsonl / test.jsonl with relative image paths — the input format of prepare_data.py:

python3 examples/diffusionnft_trainer/minimax_h3/build_fl2va_jsonl.py \
    --prompt_file dancegrpo_consist-id.txt  # see "Getting the prompt file" \
    --image_dir data/flux_images/images \
    --output_dir data/flux_images \
    --test_size 128 --seed 42

Reference dataset recipe

Prompts: 27,815 English video captions from ConsisID-preview-Data, as filtered by DanceGRPO (assets/consist-id.txt)
Images: FLUX.1-dev, 400x640, 30 steps, guidance 3.5, per-index seeds
Split: seed-42 shuffle -> 27,687 train / 128 test
Convert: prepare_data.py --frame_mode first, then train with rollout.pipeline.task=fl2va and frame_indices='[0]'. The rollout pipeline LANCZOS-resizes condition images to the sampling resolution, so training at e.g. 288x464 (same ~1:1.61 aspect) works directly.

Launch

Rollout VAE and sequence parallelism

The T2VA launcher accepts ROLLOUT_USP/ROLLOUT_RING (both default 1), VAE_PATCH_PARALLEL_SIZE (default 1), VAE_PARALLEL_MODE (default tile), and VAE_USE_TILING (default False). Existing TP-only behavior is unchanged. For example, add ROLLOUT_TP=4 VAE_PATCH_PARALLEL_SIZE=4 VAE_USE_TILING=True to reuse a four-rank TP group for tiled VAE decode.

Pure Ulysses uses ROLLOUT_TP=1 ROLLOUT_USP=4 ROLLOUT_RING=1, with TEXT_ENCODER_TP=4 VAE_PATCH_PARALLEL_SIZE=4 VAE_USE_TILING=True. Local rollout subclasses account for SP ranks without patching verl; TP remains the actual engine TP degree. GPU validation is pending. H3 only permits VAE mode tile, VAE parallel size 1 or the full DiT group, and no hybrid Ulysses x Ring. Encoder TP must divide 8.

FL2VA/Ref2VA and V1 users can pass the typed Hydra fields directly; their launchers do not yet provide these environment shortcuts. See the configuration reference for allocation semantics and the native VAE’s small-tile fallback.

Text-encoder tensor parallelism

With sequence parallelism disabled, H3 NFT launchers default TEXT_ENCODER_TP to ROLLOUT_TP, forwarded as actor_rollout_ref.rollout.text_encoder_tp_size (without +). The T2VA launcher uses the full TP * USP * Ring group when SP is configured. For example, ROLLOUT_TP=4 TEXT_ENCODER_TP=4 shards the encoder across all four DiT ranks; TEXT_ENCODER_TP=1 keeps it unsharded. Use ETP=1 or the complete DiT group, not an intermediate subgroup. This applies to T2VA, FL2VA, and Ref2VA and is independent of CPU/layerwise offload.

[!IMPORTANT] The T2VA and FL2VA NFT launchers previously omitted ETP and therefore used ETP=1. They now default to full rollout TP; set TEXT_ENCODER_TP=1 to retain the previous unsharded behavior.

The legacy +actor_rollout_ref.rollout.engine_kwargs.vllm_omni.text_encoder_tp_size override remains supported. Prefer the typed field and do not set conflicting values through both paths. The fix for #563 retains ETP through CLI conversion into the fused diffusion engine’s parallel config; a CLI argument alone was not evidence that the encoder was sharded.

Actor FSDP sequence parallelism

The T2VA, FL2VA, and Ref2VA launchers accept ACTOR_SP (default 1). Set it to a divisor of the GPU count to shard each Actor’s joint text/video/audio sequence with Diffusers Ulysses context parallelism:

ACTOR_SP=2 bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_t2va_lora.sh

This uses the same standard equal-partition Ulysses path as Qwen-Image and changes Actor FSDP data parallelism to NUM_GPUS / ACTOR_SP; it does not change rollout ROLLOUT_TP or text-encoder TEXT_ENCODER_TP. ACTOR_SP must divide the model’s attention-head count (56 for the released H3 checkpoint). When a packed layout’s length is not a multiple of ACTOR_SP, the adapter appends padding rows and masks them as attention keys; real rows attend exactly as without padding and padding outputs are discarded, so arbitrary prompt and reference lengths are supported. Use an attention backend that accepts key masks under context parallelism (native, flash_varlen_hub, or _flash_3_varlen_hub). Keep timestep staging disabled. DiffusionNFT retains its per-sample H3 forward loop, so SP reduces the memory of each compatible long packed sequence but does not combine heterogeneous samples into one transformer call.

T2VA

export MODEL_PATH=/path/to/MiniMax-H3
export DATA_DIR=/path/to/h3_t2va_data

bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_t2va_lora.sh

MiniMax H3 t2va requires an explicit named aspect_ratio (one of 21:9/16:9/4:3/1:1/3:4/9:16); the launch script sets 16:9 and explicit height/width control the actual canvas (must be multiples of 32).

The H3-specific agent loop (minimax_h3_diffusion_single_turn_agent) is required: it tokenizes raw text once and sends those token IDs directly to the H3 text encoder.

Training rollouts sample with INFER_STEPS=10 diffusion steps for throughput; validation always uses 40. Raise INFER_STEPS (e.g. 50) for higher-quality rollouts. Common overrides:

NUM_GPUS=8 ROLLOUT_TP=4 ROLLOUT_N=4 INFER_STEPS=50 \
TOTAL_TRAINING_STEPS=100 OUTPUT_DIR=/path/to/output \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_t2va_lora.sh

FL2VA (image-conditioned) training

This recipe trains the FL2VA checkpoint with online DiffusionNFT. The rollout uses vLLM-Omni’s official first/last-frame contract and the Actor applies the NFT forward-process objective only to generated video/audio rows.

Data

Prepare train.jsonl and test.jsonl. Each row has a prompt and either an images list or explicit first/last names:

{"prompt":"A sunrise becomes a starry night.","images":["images/first.png","images/last.png"]}

Convert it with:

python examples/diffusionnft_trainer/minimax_h3/prepare_data.py \
  --input_dir /path/to/raw \
  --output_dir /path/to/parquet \
  --frame_mode first_last

frame_mode can be first, last, or first_last. Set the matching launcher value:

export FRAME_INDICES='[0,-1]'  # '[0]' or '[-1]' for one-image datasets

Checkpoint

MODEL_PATH is a single MiniMax-H3 repo root that already ships both transformer layouts as siblings: FL2VA/ (the fused QKV+GEGLU rollout checkpoint used by vLLM-Omni, with rollout weights under FL2VA/transformer/) and transformer/ (the diffusers MiniMaxH3Transformer3DModel used for FSDP actor training). The official MiniMaxAI/MiniMax-H3 repo provides both, so no manual conversion is needed. Do not overwrite FL2VA/transformer/ with the diffusers transformer/: that silently breaks the rollout weight loader.

Run

MODEL_PATH=/path/to/MiniMax-H3 \
DATA_DIR=/path/to/parquet \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_fl2va_lora.sh

The latest vLLM-Omni contract requires 4–15 seconds at 24 FPS. The launcher’s NUM_FRAMES=96 is aligned by vLLM-Omni to the next valid 17n+5 boundary. Sampling edges must be multiples of 32 (the H3 pipeline silently floors anything else); the recipe uses 288x448 for training and 576x928 for validation. Training rollouts sample with INFER_STEPS=10 diffusion steps for throughput and validation uses 40; 2–4 steps are suitable only for contract smoke tests. Do not use the old short 22/29-frame settings.

To verify that the two checkpoint directories represent exactly the same base policy after fused-QKV and GEGLU conversion, run:

python tests/special_e2e/minimax_h3_checkpoint_parity.py \
  --vllm-transformer "$MODEL_PATH/FL2VA/transformer" \
  --diffusers-transformer "$MODEL_PATH/transformer"

The H3-specific agent loop is required: it tokenizes raw text once and lets vLLM-Omni prepend the <Picture N> vision presentation. Replacing it with the generic diffusion agent loop changes the prompt contract.

CPU offload is enabled and the default rollout TP is 4: with TP=2, the colocated dummy-load phase places a full text encoder beside one DiT shard before offload activates and leaves no room for Actor-to-rollout weight synchronization on a 96 GB GPU. With the 8-GPU recipe, ROLLOUT_N=4 also makes the two-prompt Actor batch divisible by the FSDP data-parallel world size.

The launcher defaults to FlashAttention 3 for both the Actor (_flash_3_varlen_hub) and the rollout (FLASH_ATTN_3_HUB), which requires the kernels package. Override ACTOR_ATTN_BACKEND and ROLLOUT_ATTN_BACKEND together (e.g. native / TORCH_SDPA) only after validating the replacement backends.

Rollout quantization is intentionally not enabled. On the pinned vLLM-Omni commit, a native custom pipeline combined with online FP8 can hit a meta-tensor placement failure during custom-pipeline initialization; BF16 TP=4 is the validated path.

Ref2VA (multi-reference) training

Ref2VA trains against any mix of image, video and standalone-audio references (up to twelve files per row: at most nine images, three videos and three audios). The rollout uses the official Ref2VA/ partition and the Actor loads the matching Diffusers weights from transformer_ref/. Reference rows remain fixed while DiffusionNFT noises and trains only the generated video/audio rows.

Data

Prepare train.jsonl and test.jsonl. Each row needs at least one image or video reference; references may be given as the plural images/videos/audios lists or the singular image/video/audio keys, and videos accept an optional start_time_seconds:

{"prompt":"Turn the reference character toward the camera.","images":["images/a.png","images/b.png"]}
{"prompt":"Continue the clip with the same voice.","videos":[{"path":"clips/ref.mp4","start_time_seconds":1.5}],"audios":["audio/voice.wav"]}

Convert the splits to parquet:

python examples/diffusionnft_trainer/minimax_h3/prepare_ref2va_data.py \
  --input_dir <raw-data-directory> \
  --output_dir <parquet-directory>

Reference layouts may vary from row to row. The Agent Loop validates each reported row count, pads video and audio condition rows to MAX_PROMPT_EMBEDS, and emits validity masks so the standard Agent Loop manager can concatenate outputs from every rollout worker. Before Actor training, the trainer converts the padded rows to jagged tensors; the DiffusionNFT engine then restores only the largest row count needed by each minibatch. Invalid shapes, inconsistent counts or masks, and rows exceeding the configured limit fail instead of being silently truncated.

Checkpoint and run

MODEL_PATH must point to the official MiniMax-H3 repository root containing both Ref2VA/ and transformer_ref/:

MODEL_PATH=<MiniMax-H3-root> \
DATA_DIR=<parquet-directory> \
REF_IMAGE_SHORT_EDGE=512 \
VAL_REF_IMAGE_SHORT_EDGE=1024 \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_ref2va_lora.sh

The H3 Agent Loop keeps the user prompt token-ID-native while vLLM-Omni adds the reference presentation. The default MAX_PROMPT_EMBEDS=12288 is both the prompt-embedding limit and the fixed transport limit for each video/audio condition-row tensor; it must cover the largest reference layout in the dataset. Global transport padding is removed before Actor minibatching, so it does not become the model’s effective sequence length. REF_IMAGE_SHORT_EDGE configures training, while VAL_REF_IMAGE_SHORT_EDGE configures validation and defaults to the training value. Both accept multiples of 32 from 256 through 2048. Reduce MAX_PROMPT_EMBEDS only after checking both the final prompt embedding and condition-row counts for the dataset. TEXT_ENCODER_TP defaults to ROLLOUT_TP so the Qwen3-VL encoder is sharded instead of residing on one DiT rank; supported values are 1 and ROLLOUT_TP.

The initial recipe reuses CLAP and ImageBind to validate joint audio-video training. These rewards do not measure similarity to the reference image, so a separate reference-aware reward is required before evaluating reference fidelity.

T2VA performance reference

The run below trains the rank-64 LoRA with the default CLAP + ImageBind multi-reward; the weighted-sum reward rises steadily as audio-video alignment improves.

Train Reward

Train reward

Train reward (detail)

Eval Reward

Eval reward

Time consumption

Time consumption

FL2VA performance reference

The FL2VA (image-conditioned) run uses the same rank-64 LoRA and CLAP + ImageBind multi-reward as T2VA and follows the same layout as above.

Train Reward

Train reward

Train reward (detail)

Eval Reward

Eval reward

Time consumption

Time consumption

License

Prompts: CC-BY-4.0 (ConsisID-preview-Data)
Images: generated with FLUX.1-dev (non-commercial license); datasets built with this pipeline inherit the non-commercial restriction

MiniMax H3 T2VA, FL2VA, and Ref2VA DiffusionNFT

Video Examples

Install

Checkpoint

Data Preparation

T2VA (prompt-only)

FL2VA first-frame data

Pipeline

Getting the prompt file

Scripts

gen_flux_images.py

build_fl2va_jsonl.py

Reference dataset recipe

Launch

Rollout VAE and sequence parallelism

Text-encoder tensor parallelism

Actor FSDP sequence parallelism

T2VA

FL2VA (image-conditioned) training

Data

Checkpoint

Run

Ref2VA (multi-reference) training

Data

Checkpoint and run

T2VA performance reference

Train Reward

Eval Reward

Time consumption

FL2VA performance reference

Train Reward

Eval Reward

Time consumption

License

`gen_flux_images.py`

`build_fl2va_jsonl.py`