Performance Tuning Guide

Last updated: 10/17/2026

This page is the starting point for tuning a VeRL-Omni diffusion RL run. It does not repeat the detail already covered by the more specific pages — instead it gives you a decision order and links to the right page for each decision, plus a troubleshooting checklist for the OOM/throughput problems that come up across all of them. For omni-modality (AR) runs, start with section 5.

Where time actually goes

A FlowGRPO-style step has three stages, and each has its own tuning surface:

Stage

What it does

Tune with

Rollout

Generate images/video/audio for each prompt

Diffusion Rollout Batching

Reward

Score generated samples

Async Reward

Actor

Compute advantages and update the policy

Tuning and Improving MFU

Before changing any config, profile the step to see which stage actually dominates wall time — see Profiling FlowGRPO / diffusion training. Guessing which stage is slow from symptoms alone is unreliable: a rollout that looks slow is often actually reward-bound once you profile it (see Async Reward).

For a live view of where time goes across a run — instead of a one-off profiler trace — see Monitor Training with RL-Insight, which surfaces trainer, rollout, and TransferQueue metrics in Grafana dashboards.

1. Decide your GPU layout first

Layout changes are the highest-leverage tuning decision and should come before any per-stage knob, because they change what “optimal” means for the other stages.

  • Colocated (actor, rollout, and reward share the same GPU pool): simplest setup, no idle GPUs, but reward/rollout and actor training time-share the same devices — a slow reward model stalls the whole step. This is the default in most examples/ scripts.

  • Disaggregated reward pool (reward.reward_model.enable_resource_pool=True): puts reward-model inference on its own GPUs so it overlaps with rollout generation instead of blocking it. Worth it once reward scoring is a significant fraction of step time — see Async Reward for the config and the GPU-count tradeoff.

Scaling to more nodes is a separate, orthogonal decision — colocated vs. disaggregated reward still applies once you add nodes. The default multi-node recipe in Multi-Node Training keeps reward colocated (REWARD_TP=4 on every node); that guide also shows how actor, rollout, and reward pools map onto nodes for both layouts.

Re-profile after any layout change — the stage that was the bottleneck before a layout change is often not the bottleneck after it.

2. Tune rollout throughput

Once the GPU layout is fixed, the rollout engine has its own batching tradeoff that is independent of everything else: step-wise continuous batching vs. request-level batching. See Diffusion Rollout Batching for how to choose between them, the config knobs (step_execution, actor_rollout_ref.rollout.max_num_seqs, ++actor_rollout_ref.rollout.engine_kwargs.vllm_omni.request_batch_max_wait_ms), and measured before/after numbers for the example recipes. max_num_seqs is the top-level engine concurrency knob in both modes; only request_batch_max_wait_ms goes under engine_kwargs.vllm_omni.

3. Tune actor throughput and memory

Tuning and Improving MFU is the actor-side playbook: param_offload / optimizer_offload, Ulysses sequence-parallel size, micro-batch size, layered_summon, and the gradient-checkpointing MFU caveat. Read that section before changing actor config — it also explains why each knob helps, which matters when your OOM point differs from the reference 20B-on-H200 setup it was written against.

4. Troubleshooting checklist

Symptoms that show up regardless of which stage causes them:

OOM during rollout generation

  • Lower max_num_seqs first if you are on the request-level batching path — it packs one full activation buffer per concurrent request rather than reusing a step-wise buffer, so it uses more memory per concurrent request for the same value (see the warning in Diffusion Rollout Batching).

  • Check actor_rollout_ref.rollout.gpu_memory_utilization isn’t already near 1.0 — this is the vLLM-Omni engine’s memory reservation, not an LLM KV cache, and it leaves less headroom for the packed activation memory a high max_num_seqs needs.

OOM during actor forward/backward

  • Decrease ppo_micro_batch_size_per_gpu first — this lowers activation memory without changing the effective batch size (see the FAQ table in FlowGRPO Quickstart).

OOM during update_actor / update_weights after micro-batching is already tight

  • This is an optimizer/parameter-state OOM, not an activation OOM — go through the offload ordering in Tuning and Improving MFU (optimizer_offload=True first, then param_offload=True as a last resort), since offloading costs less throughput than a smaller micro-batch once activation memory is no longer the bottleneck.

  • If both offload flags are already True, confirm layered_summon=True — disabling it under offload tends to OOM during weight sync.

Step time dominated by data transfer between worker groups (video models)

  • This shows up as idle GPU time between rollout and training rather than inside either stage’s compute, and is more likely on video/audio generation where trajectories are larger than for image models. It’s a V1 trainer (TransferQueue + ReplayBuffer) concern — see Diffusion V1 training for sync vs. separate_async mode and how the replay buffer moves trajectories into the training loop.

  • Trim the replay buffer’s polling cadence with trainer.v1.sampler.poll_interval (default null: 0.05 in sync mode, 2.0 in separate_async): each wait loop polls TransferQueue for finished prompt groups, so a large interval adds dead wall-clock time after rollout lands, and a very small one adds metadata syncs. Halving it typically pays off when parameter_sync_step > 1 because every local update waits once.

  • Check what actually crosses TransferQueue per step. The rollout dump read is field-projected (latents/embeds stay in TransferQueue); validation deliberately stays a full read because the colocated reward model receives that DataProto and may read any pipeline-emitted field. Trajectory writes are batched per prompt group; if you add custom reads, pass select_fields rather than fetching full rows.

Step time dominated by reward scoring

  • Confirm with a profiler trace (recipe 6 in profiler.md) before changing anything — reward cost is easy to misattribute to rollout because both run inside the same wall-clock “generation” window when reward is colocated.

  • Move to a disaggregated reward pool (reward.reward_model.enable_resource_pool=True) so scoring overlaps generation instead of gating it; see Async Reward.

Reward or optimizer step is CPU-bound and much slower than a standalone benchmark

  • Check the CPU threads per worker. Uncapped, PyTorch starts one intra-op thread per physical core in every Ray worker, so colocated reward, actor, and rollout workers oversubscribe the host. Reward-model preprocessing (PIL resize, normalize) and the CPU-offloaded optimizer step are hit hardest. Symptoms: a reward worker at 1000-2000% CPU, and per-chunk scoring time that varies 2-3x between steps.

  • Cap the threads before launch, e.g. export OMP_NUM_THREADS=8. Workers of a local Ray cluster inherit the launcher’s environment, or pass it explicitly with +ray_kwargs.ray_init.runtime_env.env_vars.OMP_NUM_THREADS="'8'" (Ray requires string values).

  • Measured on a 224-vCPU, 4-GPU host with the Boogu-Image HPSv2.1 + PickScore + CLIPScore NFT recipe (384 images/step): the reward stage fell from 87-157 s to ~4 s, update_actor from ~337 s to ~320 s, and the V1 sync step from 505-604 s to 408-423 s. V0 gains the same.

MFU looks implausibly low or above 1.0

  • See How FLOPs are computed first — if perf/mfu/actor > 1.0, the two documented causes are a mis-identified device peak on relabeled SKUs (pin the real peak with VERL_OMNI_DEVICE_FLOPS_TFLOPS) and a missing DP gather of sequence lengths, not the LoRA vs. full-FT FLOPs caveat — that one only matters when comparing LoRA against full FT, not as a path to MFU above 1.0.

5. Tuning omni-modality (AR) runs

The sections above are written against diffusion RL, where the rollout engine generates images/video/audio with a diffusion pipeline. Omni-modality runs (Qwen3-Omni thinker with GSPO/DAPO, Qwen3-TTS with GRPO) use the main_omni entrypoint with an autoregressive vLLM-Omni rollout, and the tuning surfaces differ accordingly:

  • Layout. Colocated vs. disaggregated reward still applies, but AR runs add the separate-async option for the actor/rollout split itself — Separate-Async Omni Training documents that layout, when to prefer it, and how to balance the two pools. The megatron separate-async recipes in the Qwen3-Omni README are the reference setups (experimental; see the reproducibility notes in models catalogue).

  • Rollout. AR rollout has no denoising-step knob; concurrency is governed by the vLLM-Omni engine (max_num_seqs, engine_kwargs.vllm_omni.*) — the same knobs introduced in step 2 above, minus the diffusion-specific batching modes. Diffusion Rollout Batching remains the authoritative description of those two modes.

  • Actor. The FSDP knobs documented in Tuning and Improving MFU (offload flags, sequence parallelism, micro-batch size) apply to the omni actor as well, but the reference measurements on that page are diffusion-specific; the omni recipes in models catalogue list their own validated GPU configs (H100/H200, plus NPU variants).

  • Reward. Text and multimodal reward servers are profiled and scaled the same way as diffusion reward pools — see Async Reward and HTTP Scorer.

  • Monitoring and profiling. RL-Insight works for main_omni runs. For step-based profiling, omni PPO V1 is supported while diffusion V1 is not — see the caveat in profiler.md.

The only measured omni throughput reference points are in Separate-Async Omni Training — the MiniCPM-o 4.5 colocated-vs-separate-async split ladder on 4 × 80 GB; beyond that, treat every knob above as requiring measurement on your own workload before drawing conclusions.

See also