Qwen3-Omni Thinker GSPO Trainer

Last updated: 09/09/2026

This example shows how to post-train the Qwen3-Omni-30B-A3B Thinker with GSPO on multimodal reasoning tasks, using FSDP for the actor and vllm-omni as the async rollout backend. Three input recipes are supported: text → text (gsm8k), image → text (MMK12), and text + image + audio → text (AVQA-R1-6K).

Both GPU and NPU training platforms are supported:

For the base environment setup, see the installation guide.

Installation

Follow the installation guide to set up the base environment. In short:

git clone https://github.com/verl-project/verl-omni.git && cd verl-omni
uv venv --python 3.12 --seed && source .venv/bin/activate
uv pip install -e ".[gpu]" --torch-backend=auto
uv pip install "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@$(cat .github/vllm_omni_pin.txt)"
uv pip install -e ".[train,dev]"
# flash-attn is required for GPU training
uv pip install flash-attn>=2.8.3

Tested with transformers==5.13.1, accelerate==1.14.0, peft==0.19.1.

Verify:

python -c "import verl, verl_omni, vllm, vllm_omni; print('OK')"

The GPU V1 and AVQA NPU launchers use verl_omni.trainer.main_omni and set VERL_USE_EXTERNAL_MODULES=verl_omni. Processor/model setup is handled by the registered Qwen3-Omni V1 adapter, so these launchers do not load model monkey-patches through external_lib.

The launchers colocate the FSDP actor and the vllm-omni rollout on the same devices. run_qwen3_omni_thinker_gspo_lora_v1.sh targets a single node with 4 × H800 80GB; run_qwen3_omni_thinker_gspo_npu_avqa_v1.sh targets a single Atlas 800T A3 node with 16 × Ascend 910C 64GB. It dynamically generates a thinker-only deploy config for each rollout replica from that replica’s visible devices, avoiding cross-replica device-rank collisions.

Prepare the model

The GPU V1 scripts default MODEL_PATH to $HOME/models/Qwen/Qwen3-Omni-30B-A3B-Instruct (~60 GB). The NPU script defaults to the HuggingFace Hub ID Qwen/Qwen3-Omni-30B-A3B-Instruct. To use a different local copy or Hub ID, set MODEL_PATH:

export MODEL_PATH=/path/to/local/Qwen3-Omni-30B-A3B-Instruct

Training with gsm8k

Prepare the dataset

A parquet dataset of GSM8K math problems, defaulting to ~/data/gsm8k/{train,test}.parquet. Use verl’s gsm8k.py converter:

python gsm8k.py --local_save_dir ~/data/gsm8k
ls ~/data/gsm8k/   # train.parquet  test.parquet

Run training

Launch from the repository root — pick the flavor that matches your hardware:

# GPU, LoRA (r=32), 4 × H800 — V1 trainer (recommended)
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh

# NPU, AVQA, Atlas 800T A3 (16 × Ascend 910C 64GB) — V1 trainer
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_npu_avqa_v1.sh

The V1 launchers use pure CLI overrides on verl_omni.trainer.main_omni (no --config-path/--config-name, no recipe YAML). Config precedence, lowest to highest:

verl omni_trainer defaults  →  CLI overrides (run script)  →  "$@" extra args

Any field can be overridden from the command line without editing the script:

MODEL_PATH=/local/Qwen3-Omni-30B-A3B-Instruct \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_v1.sh \
    trainer.total_epochs=10 \
    actor_rollout_ref.actor.optim.lr=2e-6

What is trained

Only the Thinker (Qwen3OmniMoeThinkerForConditionalGeneration):

  • GPU (LoRA) — rank 32, alpha 64, on target_modules="['q_proj','k_proj','v_proj','o_proj']" (the V1 Qwen3OmniThinkerAdapter.configure_model handles Thinker-forward redirection and _verl_strip_modules via get_strip_modules, so exclude_modules only needs to cover the heads/encoders).

  • NPU (full-parameter) — LoRA is disabled (lora_rank=0); all Thinker parameters are updated under FSDP.

  • exclude_modules strips talker / code2wav / code_predictor / visual / audio_tower; freeze_vision_tower=True keeps the vision encoder cold.

  • configure_model in the registered adapter (verl_omni/pipelines/qwen3_omni/thinker_training_adapter.py) redirects module.forward → module.thinker.forward and sets _no_split_modules after the default base-class stripping.

Reward comes from the naive reward manager (math accuracy on parsed answers).

Healthy signals (gsm8k, 4×H800, LoRA r=32):

  • training/rollout_actor_probs_pearson_corr > 0.995 (actor ↔ rollout agree after weight sync) — the primary correctness signal.

  • rollout_corr/log_ppl_diff ≈ 0.001 (near zero, confirms rollout↔actor log-prob consistency).

  • actor/loss ≈ 1e-5, actor/grad_norm ∈ [1e-3, 1e-2], no OOM (actor/perf/max_memory_allocated_gb < 45).

  • val-core/openai/gsm8k/acc/mean@1 rising with steps.

Training with MMK12

For visual math reasoning we ship an end-to-end pipeline on top of the MMK12 dataset (image input + text output, K12 math). It reuses the same GSPO recipe as the text-only path — only the data preprocessing and the reward scorer differ. Use the dedicated V1 GPU/LoRA script:

Prepare the dataset

Download the raw MMK12 parquet shards (from ModelScope or HuggingFace) into a local directory — the loader expects filenames like train-*.parquet and test-*.parquet — and convert them into the verl RL parquet layout with:

python examples/gspo_trainer/data_process/mmk12.py \
    --local_dataset_path /path/to/mmk12/ \
    --local_save_dir ~/data/mmk12

The converter emits one verl RL row per problem, with data_source="math_dapo", a system prompt that constrains the model to emit <answer>…\boxed{…}…</answer>, and the image bytes carried inline in the images column so the parquet stays self-contained. Input / kept / dropped counts and answer-type tallies are printed at the end. See the module docstring in examples/gspo_trainer/data_process/mmk12.py for the exact output schema.

Run training

The MMK12 reward scorer grades responses with math_verify. Multimodal data processing also requires qwen-vl-utils for vision info extraction. Install both explicitly:

pip install math-verify qwen-vl-utils

Then launch the MMK12 V1 training script:

TRAIN_FILE=$HOME/data/mmk12/train.parquet \
VAL_FILE=$HOME/data/mmk12/test.parquet \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh

For Ascend NPU training, use the NPU variant:

TRAIN_FILE=$HOME/data/mmk12/train.parquet \
VAL_FILE=$HOME/data/mmk12/test.parquet \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1_npu.sh

Override the model, dataset, or MMK12 reward scorer path without editing the script:

MODEL_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \
TRAIN_FILE=/path/to/train.parquet \
VAL_FILE=/path/to/test.parquet \
REWARD_FUNCTION_PATH=/path/to/custom_reward.py \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1_npu.sh

Compared with the GPU script, the NPU variant includes two important Ascend settings:

  • export VLLM_ASCEND_ENABLE_NZ=0 disables the NZ format in vLLM Ascend.

  • actor_rollout_ref.rollout.cudagraph_capture_sizes limits the graph shapes captured by the rollout engine. Capturing too many shapes can cause runtime errors, so keep this list sparse. The current script uses capture sizes [1,2,4,16,64,128,512,1024,2048,3072,4096]. The script registers the custom reward scorer internally (no yaml edits required). Override LR or other fields via “$@” extras:

TRAIN_FILE=$HOME/data/mmk12/train.parquet \
VAL_FILE=$HOME/data/mmk12/test.parquet \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh \
    actor_rollout_ref.actor.optim.lr=3e-6

The scorer combines math_verify accuracy with a progressive format reward on the <answer>…\boxed{}…</answer> template; see verl_omni/utils/reward_score/mmk12_reward.py for the full formula.

MMK12 On-Policy Distillation (OPD)

OPD distills a teacher’s distribution into the student during GSPO training. The student is the noised Qwen3-Omni-30B-A3B-Instruct (25% weight noise) and the teacher is the original (un-noised) model, served by vllm_omni in AR mode.

Validated on 2 × Ascend 910C machines — student rollout/actor on 16 GPUs of node 1, teacher model on 16 GPUs of node 2:

# 1. On the master node (node 1): ray start --head
# 2. On the slave node (node 2): ray start --address='<head_ip>:<port>'
# 3. Run on the master node:
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_mmk12_v1_opd_npu.sh

The script sets distillation.enabled=true with a vllm_omni teacher (loss_mode=kl, use_policy_gradient=true). Teacher and student must share the same tokenizer (same model family).

Training with AVQA-R1-6K

The AVQA recipe trains the Qwen3-Omni Thinker to answer a four-way question from question text, one image, and one WAV clip. The output is text ending in a single option tag such as <answer>B</answer>.

Prepare the dataset

python examples/gspo_trainer/data_process/avqa.py \
    --input_dir /path/to/AVQA_R1 \
    --output_dir ~/data/avqa_r1_6k

This writes train.parquet and validation.parquet. The parquet stores absolute image/audio paths, so the AVQA media directory must be mounted at the same path on every Ray worker. The converter validates modalities, options, labels, and media existence and prints kept/dropped counts for each split.

Image and audio paths are decoded by Qwen’s qwen_omni_utils.process_mm_info through QwenOmniRLHFDataset. Install the official media loader without changing the NPU engine stack with pip install -e ".[audio]". ffmpeg is only required when the dataset carries compressed audio (mp3/m4a/aac/ogg) or http(s) audio URLs — those go through audioread/ffmpeg. Plain local WAV files decode via librosa/soundfile (libsndfile) and need no ffmpeg.

Run GPU training

Launch the GPU LoRA script (4 × H800 80GB, LoRA r=32, same GSPO recipe as the other GPU recipes). The audio-specific settings it adds on top of the base recipe are:

  1. data.custom_cls = QwenOmniRLHFDataset — the audio-aware dataset class that parses <audio> placeholders and loads the WAV files (the default dataset handles images only).

  2. +data.mm_processor_kwargs.sampling_rate=16000 — the Qwen3-Omni feature extractor rate, used when filtering overlong multimodal prompts.

  3. Rollout memory-margin knobs for colocated audio workloads — gpu_memory_utilization=0.7 (the base GPU recipe uses 0.8), rollout.prompt_length=4160, engine_kwargs.vllm_omni.max_num_seqs=256, and cudagraph_capture_sizes=[1,2,4,8,16,32,64,128,256] — see Sizing rollout memory in colocated sleep mode in the integrating guide.

bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_lora_avqa_v1.sh

The reward extracts the first <answer>...</answer> payload from the response and returns a binary exact-match score against the tagged dataset label.

Run NPU training

Use the dedicated V1 AVQA NPU launcher. It uses FSDP2 with CPU offload, a 16-NPU topology, rollout TP=4, and four rollout workers without changing the existing generic NPU script.

TRAIN_FILE=$HOME/data/avqa_r1_6k/train.parquet \
VAL_FILE=$HOME/data/avqa_r1_6k/validation.parquet \
MODEL_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \
bash examples/gspo_trainer/qwen3_omni/run_qwen3_omni_thinker_gspo_npu_avqa_v1.sh

The launcher uses a 4096-token multimodal prompt budget, a 12288-token response budget, and 128 prompts with 16 responses each per rollout. It trains for 10 epochs, caps dynamic actor and log-prob batches at 20480 tokens per NPU, and computes entropy in 2048-token chunks to reduce peak NPU memory. It registers the audio-aware dataset class by importable package path so multiprocessing preserves its RLHFDataset base class, sets rollout NPU memory utilization to 0.6, uses deterministic validation, and wires choice_reward.py.

Performance

All GPU results measured on a single node of 4 × H800 80GB, actor and rollout colocated, LoRA r=32, GSPO. Curves are hosted in the shared verl-omni/gspo_demo W&B project; the runs were trained with the release/v0.2.0 branch.

Script

Dataset

# Cards

Batch × rollout.n

lr

Steps

val acc@1 / reward@1

rollout↔actor pearson

GPU memory

gsm8k (wandb)

gsm8k

4

128 × 16 = 2048

3e-6

578

acc 0.971

0.998

~43 GB

MMK12 (wandb)

MMK12

4

128 × 16 = 2048

3e-6

392

reward 0.811

0.998

~59 GB

AVQA-R1-6K (wandb)

AVQA-R1-6K

4

128 × 16 = 2048

3e-6

348

reward 0.877

0.996

~46 GB

gsm8k (wandb, naive reward, math accuracy): critic/rewards/mean rose from ~0.88 to ~0.97, val-core/openai/gsm8k/acc/mean@1 reached 0.971. rollout_corr/log_ppl_diff stayed near zero (~0.002).

MMK12 (wandb, composite reward, math_verify + format): critic/rewards/mean rose from ~0.71 to ~0.81, val-core/mmk12/reward/mean@1 reached 0.811 (last logged at step 392). rollout_corr/log_ppl_diff stayed near zero (~0.002).

AVQA-R1-6K (wandb, binary <answer> exact-match reward): critic/rewards/mean rose from ~0.73 to ~0.94, val-core/avqa_r1_6k/reward/mean@1 reached 0.877. rollout_corr/log_ppl_diff stayed near zero (~0.007).

Logging

W&B logging is enabled by default:

export WANDB_API_KEY=<your_wandb_api_key>
# trainer.project_name / experiment_name are already set in the script

File map

examples/gspo_trainer/
├── qwen3_omni/
│   ├── run_qwen3_omni_thinker_gspo_lora_v1.sh       ← V1 launch script (GPU, LoRA r=32, text)
│   ├── run_qwen3_omni_thinker_gspo_lora_mmk12_v1.sh  ← V1 launch script (GPU, LoRA r=32, image)
│   ├── run_qwen3_omni_thinker_gspo_lora_avqa_v1.sh   ← V1 launch script (GPU, LoRA r=32, audio + image)
│   ├── run_qwen3_omni_thinker_gspo_npu_avqa_v1.sh    ← V1 launch script (NPU, AVQA)
├── data_process/
│   ├── mmk12.py                                      ← MMK12 → verl RL parquet converter
│   └── avqa.py                                       ← AVQA → verl RL parquet converter
└── README.md                                         ← (this file)