MiniMax-H3 text-to-audio-video DiffusionNFT training

Last updated: 09/01/2026

This recipe trains a rank-64 MiniMax H3 LoRA with online DiffusionNFT for text-to-audio-video (T2VA). A Diffusers transformer is trained with FSDP2 while vLLM-Omni generates joint video and audio rollouts. CLAP and ImageBind provide the default multi-reward (audio-video alignment).

A ready-made FL2VA first-frame dataset built with the data pipeline below is published at https://huggingface.co/datasets/zyfenghit/dancegrpo-t2av

Video Examples

ID	Prompt	MiniMax H3 base model	MiniMax H3 + DiffusionNFT
1	stickman monigote shooting a energy sphere from his hands
2	a husky dog with sunglasses riding on santas sled
3	minimalist polygonal human skull in green flames with strong movement, uhd
4	17th century sailing ship making a path through the waves during a storm
5	close up of a skin texture, two hands with black gloves tatoo a red butterfly over it

Install

Follow the project installation guide, then install the repository-pinned vLLM-Omni revision:

uv pip install -e ".[gpu]" --torch-backend=auto
uv pip install "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@$(cat .github/vllm_omni_pin.txt)"
uv pip install -e ".[train,dev]"
uv pip install "diffusers @ git+https://github.com/huggingface/diffusers.git@d6726f38a0c5ca6c06a8f227fb7bade3486ed98d"

The explicit Diffusers revision is the tested API target that provides MiniMaxH3Transformer3DModel and the MiniMax H3 reference-conditioning components used by Ref2VA.

Checkpoint

MODEL_PATH must be a local MiniMax-H3 repo root containing FL2VA/ (vLLM-Omni rollout checkpoint) and transformer/ (converted Diffusers MiniMaxH3Transformer3DModel for FSDP training). Do not replace the official rollout transformer with a symlink to the Diffusers conversion.

Data Preparation

T2VA (prompt-only)

Convert prompt splits to prompt-only parquet (no condition images, and no negative prompts, since H3 is CFG-distilled):

python3 examples/diffusionnft_trainer/minimax_h3/prepare_t2va_data.py \
    --input_dir /path/to/raw_prompts \
    --output_dir /path/to/h3_t2va_data

Input is train.txt/test.txt (one prompt per line) or train.jsonl/test.jsonl (prompt/text/caption fields).

FL2VA first-frame data

Offline pipeline for MiniMax H3 FL2VA (text+image to audio-video) RL training: turn a prompt list into FLUX reference images, pair them into train/test JSONL, and feed the FL2VA prepare_data.py converter.

Pipeline

prompts.txt ──► gen_flux_images.py ──► images/{index:06d}.jpg
                     │                        │
                     │                        ▼
                     │              build_fl2va_jsonl.py
                     │                        │
                     ▼                        ▼
              (same index)          train.jsonl / test.jsonl
                                             │
                                             ▼
                              prepare_data.py --frame_mode first
                                             │
                                             ▼
                                train.parquet / test.parquet

Getting the prompt file

dancegrpo_consist-id.txt is the filtered ConsisID prompt list released by DanceGRPO (27,815 prompts, one per line). Download it directly from the DanceGRPO repository:

curl -L -o dancegrpo_consist-id.txt \
  https://raw.githubusercontent.com/XueZeyue/DanceGRPO/main/assets/consist-id.txt

The prompts originate from ConsisID-preview-Data captions; DanceGRPO ships the filtered result as-is and does not document the exact filtering criteria, so downloading the released file is the reproducible way to get the identical prompt set (verified line-for-line against the copy used for the published dataset).

Scripts

`gen_flux_images.py`

Multi-GPU FLUX batch image generator. Reads one prompt per line, shards the prompts across ranks (torchrun), and writes one JPEG per prompt plus a per-rank metadata_rankN.jsonl (image <-> prompt <-> index mapping). Deterministic per-prompt seed (seed + index), and re-running skips images that already exist, so interrupted jobs resume safely. Defaults mirror DanceGRPO’s online reference pipeline (400x640, 30 steps, guidance 3.5, max_sequence_length 512), so prompt and condition image are semantically aligned by construction.

torchrun --nproc_per_node=8 examples/diffusionnft_trainer/minimax_h3/gen_flux_images.py \
    --prompt_file dancegrpo_consist-id.txt  # see "Getting the prompt file" \
    --model_path /path/to/FLUX.1-dev \
    --output_dir data/flux_images \
    --height 400 --width 640

`build_fl2va_jsonl.py`

Pairs each prompt with its same-index image, verifies all images exist, shuffles with a fixed seed, and writes train.jsonl / test.jsonl with relative image paths — the input format of prepare_data.py:

python3 examples/diffusionnft_trainer/minimax_h3/build_fl2va_jsonl.py \
    --prompt_file dancegrpo_consist-id.txt  # see "Getting the prompt file" \
    --image_dir data/flux_images/images \
    --output_dir data/flux_images \
    --test_size 128 --seed 42

Reference dataset recipe

Prompts: 27,815 English video captions from ConsisID-preview-Data, as filtered by DanceGRPO (assets/consist-id.txt)
Images: FLUX.1-dev, 400x640, 30 steps, guidance 3.5, per-index seeds
Split: seed-42 shuffle -> 27,687 train / 128 test
Convert: prepare_data.py --frame_mode first, then train with rollout.pipeline.task=fl2va and frame_indices='[0]'. The rollout pipeline LANCZOS-resizes condition images to the sampling resolution, so training at e.g. 288x464 (same ~1:1.61 aspect) works directly.

Launch

export MODEL_PATH=/path/to/MiniMax-H3
export DATA_DIR=/path/to/h3_t2va_data

bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_t2va_lora.sh

MiniMax H3 t2va requires an explicit named aspect_ratio (one of 21:9/16:9/4:3/1:1/3:4/9:16); the launch script sets 16:9 and explicit height/width control the actual canvas (must be multiples of 32).

The H3-specific agent loop (minimax_h3_diffusion_single_turn_agent) is required: it tokenizes raw text once and sends those token IDs directly to the H3 text encoder.

Training rollouts sample with INFER_STEPS=10 diffusion steps for throughput; validation always uses 40. Raise INFER_STEPS (e.g. 50) for higher-quality rollouts. Common overrides:

NUM_GPUS=8 ROLLOUT_TP=4 ROLLOUT_N=4 INFER_STEPS=50 \
TOTAL_TRAINING_STEPS=100 OUTPUT_DIR=/path/to/output \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_t2va_lora.sh

FL2VA (image-conditioned) training

This recipe trains the FL2VA checkpoint with online DiffusionNFT. The rollout uses vLLM-Omni’s official first/last-frame contract and the Actor applies the NFT forward-process objective only to generated video/audio rows.

Data

Prepare train.jsonl and test.jsonl. Each row has a prompt and either an images list or explicit first/last names:

{"prompt":"A sunrise becomes a starry night.","images":["images/first.png","images/last.png"]}

Convert it with:

python examples/diffusionnft_trainer/minimax_h3/prepare_data.py \
  --input_dir /path/to/raw \
  --output_dir /path/to/parquet \
  --frame_mode first_last

frame_mode can be first, last, or first_last. Set the matching launcher value:

export FRAME_INDICES='[0,-1]'  # '[0]' or '[-1]' for one-image datasets

Checkpoint

MODEL_PATH is a single MiniMax-H3 repo root that already ships both transformer layouts as siblings: FL2VA/ (the fused QKV+GEGLU rollout checkpoint used by vLLM-Omni, with rollout weights under FL2VA/transformer/) and transformer/ (the diffusers MiniMaxH3Transformer3DModel used for FSDP actor training). The official MiniMaxAI/MiniMax-H3 repo provides both, so no manual conversion is needed. Do not overwrite FL2VA/transformer/ with the diffusers transformer/: that silently breaks the rollout weight loader.

Run

MODEL_PATH=/path/to/MiniMax-H3 \
DATA_DIR=/path/to/parquet \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_fl2va_lora.sh

The latest vLLM-Omni contract requires 4–15 seconds at 24 FPS. The launcher’s NUM_FRAMES=96 is aligned by vLLM-Omni to the next valid 17n+5 boundary. Sampling edges must be multiples of 32 (the H3 pipeline silently floors anything else); the recipe uses 288x448 for training and 576x928 for validation. Training rollouts sample with INFER_STEPS=10 diffusion steps for throughput and validation uses 40; 2–4 steps are suitable only for contract smoke tests. Do not use the old short 22/29-frame settings.

To verify that the two checkpoint directories represent exactly the same base policy after fused-QKV and GEGLU conversion, run:

python tests/special_e2e/minimax_h3_checkpoint_parity.py \
  --vllm-transformer "$MODEL_PATH/FL2VA/transformer" \
  --diffusers-transformer "$MODEL_PATH/transformer"

The H3-specific agent loop is required: it tokenizes raw text once and lets vLLM-Omni prepend the <Picture N> vision presentation. Replacing it with the generic diffusion agent loop changes the prompt contract.

CPU offload is enabled and the default rollout TP is 4: with TP=2, the colocated dummy-load phase places a full text encoder beside one DiT shard before offload activates and leaves no room for Actor-to-rollout weight synchronization on a 96 GB GPU. With the 8-GPU recipe, ROLLOUT_N=4 also makes the two-prompt Actor batch divisible by the FSDP data-parallel world size.

The launcher defaults to FlashAttention 3 for both the Actor (_flash_3_varlen_hub) and the rollout (FLASH_ATTN_3_HUB), which requires the kernels package. Override ACTOR_ATTN_BACKEND and ROLLOUT_ATTN_BACKEND together (e.g. native / TORCH_SDPA) only after validating the replacement backends.

Rollout quantization is intentionally not enabled. On the pinned vLLM-Omni commit, a native custom pipeline combined with online FP8 can hit a meta-tensor placement failure during custom-pipeline initialization; BF16 TP=4 is the validated path.

Ref2VA (multi-reference) training

Ref2VA trains against any mix of image, video and standalone-audio references (up to twelve files per row: at most nine images, three videos and three audios). The rollout uses the official Ref2VA/ partition and the Actor loads the matching Diffusers weights from transformer_ref/. Reference rows remain fixed while DiffusionNFT noises and trains only the generated video/audio rows.

Data

Prepare train.jsonl and test.jsonl. Each row needs at least one image or video reference; references may be given as the plural images/videos/audios lists or the singular image/video/audio keys, and videos accept an optional start_time_seconds:

{"prompt":"Turn the reference character toward the camera.","images":["images/a.png","images/b.png"]}
{"prompt":"Continue the clip with the same voice.","videos":[{"path":"clips/ref.mp4","start_time_seconds":1.5}],"audios":["audio/voice.wav"]}

Convert the splits to parquet:

python examples/diffusionnft_trainer/minimax_h3/prepare_ref2va_data.py \
  --input_dir <raw-data-directory> \
  --output_dir <parquet-directory>

Reference layouts may vary from row to row. The Agent Loop validates each reported row count, pads video and audio condition rows to MAX_PROMPT_EMBEDS, and emits validity masks so the standard Agent Loop manager can concatenate outputs from every rollout worker. Before Actor training, the trainer converts the padded rows to jagged tensors; the DiffusionNFT engine then restores only the largest row count needed by each minibatch. Invalid shapes, inconsistent counts or masks, and rows exceeding the configured limit fail instead of being silently truncated.

Checkpoint and run

MODEL_PATH must point to the official MiniMax-H3 repository root containing both Ref2VA/ and transformer_ref/:

MODEL_PATH=<MiniMax-H3-root> \
DATA_DIR=<parquet-directory> \
REF_IMAGE_SHORT_EDGE=512 \
VAL_REF_IMAGE_SHORT_EDGE=1024 \
bash examples/diffusionnft_trainer/minimax_h3/run_minimax_h3_ref2va_lora.sh

The H3 Agent Loop keeps the user prompt token-ID-native while vLLM-Omni adds the reference presentation. The default MAX_PROMPT_EMBEDS=12288 is both the prompt-embedding limit and the fixed transport limit for each video/audio condition-row tensor; it must cover the largest reference layout in the dataset. Global transport padding is removed before Actor minibatching, so it does not become the model’s effective sequence length. REF_IMAGE_SHORT_EDGE configures training, while VAL_REF_IMAGE_SHORT_EDGE configures validation and defaults to the training value. Both accept multiples of 32 from 256 through 2048. Reduce MAX_PROMPT_EMBEDS only after checking both the final prompt embedding and condition-row counts for the dataset. TEXT_ENCODER_TP defaults to ROLLOUT_TP so the Qwen3-VL encoder is sharded instead of residing on one DiT rank; supported values are 1 and ROLLOUT_TP.

The initial recipe reuses CLAP and ImageBind to validate joint audio-video training. These rewards do not measure similarity to the reference image, so a separate reference-aware reward is required before evaluating reference fidelity.

T2VA performance reference

The run below trains the rank-64 LoRA with the default CLAP + ImageBind multi-reward; the weighted-sum reward rises steadily as audio-video alignment improves.

Train Reward

Train reward

Train reward (detail)

Eval Reward

Eval reward

Time consumption

Time consumption

FL2VA performance reference

The FL2VA (image-conditioned) run uses the same rank-64 LoRA and CLAP + ImageBind multi-reward as T2VA and follows the same layout as above.

Train Reward

Train reward

Train reward (detail)

Eval Reward

Eval reward

Time consumption

Time consumption

License

Prompts: CC-BY-4.0 (ConsisID-preview-Data)
Images: generated with FLUX.1-dev (non-commercial license); datasets built with this pipeline inherit the non-commercial restriction

MiniMax-H3 text-to-audio-video DiffusionNFT training

Video Examples

Install

Checkpoint

Data Preparation

T2VA (prompt-only)

FL2VA first-frame data

Pipeline

Getting the prompt file

Scripts

gen_flux_images.py

build_fl2va_jsonl.py

Reference dataset recipe

Launch

FL2VA (image-conditioned) training

Data

Checkpoint

Run

Ref2VA (multi-reference) training

Data

Checkpoint and run

T2VA performance reference

Train Reward

Eval Reward

Time consumption

FL2VA performance reference

Train Reward

Eval Reward

Time consumption

License

`gen_flux_images.py`

`build_fl2va_jsonl.py`