Named Reward Models
Last updated: 11/13/2026
This guide describes how to configure and extend named model-backed rewards
under reward.models in verl-omni. For the general Reward Loop interface and
custom reward functions, refer to the upstream verl documentation.
reward.models lets one training job use one or more independently managed
model-backed rewards. It supports an engine backend, a native backend, or both
in the same job.
The framework deliberately separates inference from scoring:
a named model owns resources, inference access, and lifecycle;
a reward function converts one training sample into model inputs and converts the model output into a score;
MultiRewardManagerowns model-aware scorer dispatch, per-term outputs, failure handling, and weighted aggregation without owning an input modality.An input-specific subclass prepares scorer arguments. The current named-model recipes use the visual contract supplied by
MultiVisualRewardManager.
PickScore is an example of this contract, not a special case in the framework.
Named models require a MultiRewardManager subclass with an input contract.
Current maintained recipes use the visual subclass explicitly; the framework
does not rewrite a user-provided manager:
reward:
reward_manager:
name: MultiVisualRewardManager
Consolidating audio, text, and other input contracts behind the shared manager is separate follow-up work. Until then, modality-specific managers and their existing recipes remain unchanged.
Native replica scheduling
Named models use static padded splitting by default: each worker receives one
equal-sized chunk. For native models whose sample costs or replica speeds vary,
set reward.models.<name>.dispatch_batch_size to a positive integer to enable
completion-driven dispatch:
reward:
models:
pickscore:
backend: native
placement:
devices: [0, 1]
executor:
model: verl_omni.utils.reward_score.pickscore_reward:PickScoreNativeModel
dispatch_batch_size: 8
Keep the model’s executor arguments and reward function configured as described
below. Each placement entry still owns one complete model replica. Each replica
receives at most one scoring RPC at a time and takes the next contiguous batch
when it finishes. The final batch may be smaller; no duplicate padding is added.
Results are restored to input order before existing per-model score aggregation.
Omit this field or set it to null to keep static splitting. Engine models reject
this option; their internal scheduling and parallelism remain engine-owned.
Choose a batch size large enough for efficient model batching but small enough
to leave work available for faster replicas. It caps the in-flight samples per
replica, so it also limits how many single-sample requests can be waiting for a
model consumer. For PickScore, whose native consumer caps each forward at 16,
one inference per sample and dispatch_batch_size: 8 can coalesce at most eight
requests. Values above 16 can reduce RPC count but cannot raise that forward
batch cap; an actual forward can still be smaller. Opt in only when samples can
be scored independently: batch-sensitive or replica-local random scorers can
change scores when batch boundaries or replica assignments change.
On a dispatched scoring failure or caller cancellation, no new batches are submitted once the dispatcher observes it. Already submitted RPCs are drained before the error is raised and models sleep, including across mixed engine/native groups. There is no automatic retry, actor recovery, speculative execution, autoscaling or streaming-trainer support. A stuck RPC can therefore delay drain; this option does not introduce a timeout or health-check policy.
Backend selection
Backend |
Use it when |
Reward-function arguments |
|---|---|---|
|
The model is supported by the existing |
|
|
The model can be loaded and called directly in a reward worker, including ordinary Transformers models |
|
The engine path documented here uses vLLM. vLLM-Omni reward serving is not implemented; vLLM-Omni can still be used independently for actor rollout.
Existing jobs without reward.models continue to use the existing single-model
reward path. Do not set reward.reward_model.enable=true together with
reward.models.
Migrate an existing engine reward
An existing single-model engine configuration has one global reward model and one custom reward function:
reward:
reward_model:
enable: true
enable_resource_pool: true
n_gpus_per_node: 2
nnodes: 1
model_path: /models/ocr
rollout:
name: vllm
tensor_model_parallel_size: 2
custom_reward_function:
path: pkg://my_package.ocr_reward
name: compute_score
To migrate it, disable the existing single model, create a named engine model,
and move the scoring function into reward.reward_functions. The model name and
reward term name can be the same:
reward:
reward_model:
enable: false
enable_resource_pool: true
n_gpus_per_node: 2
nnodes: 1
models:
ocr:
backend: engine
model_path: /models/ocr
n_gpus_per_node: 2
nnodes: 1
rollout:
name: vllm
tensor_model_parallel_size: 2
reward_functions:
ocr:
path: pkg://my_package.ocr_reward
name: compute_score
weight: 1.0
required: true
Remove the old custom_reward_function.path override when switching to
reward_functions. The score function can keep the existing router-based
contract if it already accepts reward_router_address and model_name:
async def compute_score(
data_source,
solution_image,
ground_truth,
extra_info,
reward_router_address,
model_name,
):
response = await call_openai_compatible_server(
address=reward_router_address,
model=model_name,
image=solution_image,
prompt=ground_truth,
)
return {"score": parse_score(response)}
reward.reward_model.rollout remains the common engine default. Values under a
named model’s rollout override those defaults. A named model’s model_path
also overrides the common reward_model.model_path fallback.
SD3.5 V1 OCR recipe
The SD3.5 V1 synchronous recipe
uses reward.models.ocr with the engine backend and
MultiVisualRewardManager. It keeps the existing
Qwen/Qwen2.5-VL-3B-Instruct checkpoint and compute_score_ocr scorer, with
weight 1.0 and required=true.
Prepare data/ocr/sd3/train.parquet and data/ocr/sd3/test.parquet under
OCR_WORKSPACE, then run from the repository root:
OCR_WORKSPACE=/path/to/workspace bash examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora_v1.sh \
'trainer.logger=[console]'
Caller overrides remain last. This recipe opts into
reward.reward_functions.ocr.use_rollout_sampling_params=true to forward the
response length and optional deterministic seed from the model’s rollout
settings, falling back to reward.reward_model.rollout. Explicit scorer
sampling_params take precedence. Existing named rewards without this opt-in
keep their scorer defaults; OCR still defaults to 4096 output tokens and honors
GENRM_OCR_SEED. The opt-in preserves the legacy visual manager’s seed behavior:
an environment seed is excluded unless rollout determinism supplies a seed.
Model-to-reward binding
A reward term automatically uses a model with the same name:
reward:
models:
quality:
backend: native
model_path: /models/quality
placement:
devices: [0]
executor:
model: my_package.reward_model:QualityModel
reward_functions:
quality:
path: pkg://my_package.reward_score
name: compute_quality_score
Set model explicitly when the names differ or several reward terms share one
model:
reward_functions:
semantic_quality:
model: quality
path: pkg://my_package.reward_score
name: compute_semantic_quality
VLM-as-a-judge
One judge model can serve several reward terms, each with its own user_prompt
(literal text, or a pkg:// builder). For UnifiedReward, literal text may contain
{caption}, and a custom builder receives the caption. Leaving user_prompt
unset keeps the scorer’s default request.
reward_functions:
ocr:
model: judge
path: pkg://verl_omni.utils.reward_score.genrm_ocr
name: compute_score_ocr
user_prompt: "Transcribe all text in the image exactly."
semantics:
model: judge
path: pkg://verl_omni.utils.reward_score.unified_reward
name: compute_score_unified_reward
user_prompt: pkg://my_package.prompts.build_semantic_prompt
Every term uses the existing aggregation contract:
final_reward = sum(term.weight * term.score)
required=true makes a scoring failure fatal. An optional term records the
error and contributes zero. Model setup and lifecycle failures are always
fatal. ArtifactContractError is fatal for every term, including optional
terms, so missing or invalid named media cannot silently contribute zero.
Concurrent independent reward terms
By default, reward terms run in configuration order within each sample, while separate samples retain their existing concurrency. To overlap independent terms, set a worker-wide cap above one and opt in each term:
reward:
multi_reward_concurrency: 2
reward_functions:
first:
path: pkg://my_package.rewards
name: score_first
independent: true
second:
path: pkg://my_package.rewards
name: score_second
independent: true
independent must be a boolean and defaults to false. Eligible terms use
coroutine score functions with no named model (for example, an external HTTP
scorer) or an engine named model. Native model terms and synchronous score
functions cannot opt in. Opting in asserts that score functions treat sample
inputs as read-only, do not share mutable state, and have no dependencies on
other reward terms. The manager passes the same input objects to each scorer;
it does not copy large image or other payloads.
Adjacent independent terms execute in windows of at most
multi_reward_concurrency; any other term is an ordering barrier for its
sample. When the cap is above one, it bounds active independent scorer calls
across all samples handled by one reward worker. Barriers do not use the cap
and retain cross-sample concurrency, even while independent calls are active.
Each window waits for all its terms to settle before admitting the next window
for that sample, even if one term finishes and frees a semaphore permit while
another is still running. This bounds the number of admitted tasks rather than
enqueueing every term at once; a freed permit remains available to independent
terms from other samples.
Scores, diagnostics, and required/optional error handling remain in
configuration order. The cap does not limit reward workers, barrier calls, or
engine-side requests. Increasing it does not establish a throughput
improvement by itself. Cancelling a serial synchronous scorer stops its caller
without waiting for its executor thread; the thread may continue until the
scorer returns.
An admitted window settles all its terms before reporting a contract error.
Independent async scorers must cooperate with cancellation. A scorer that
swallows CancelledError can block window drain and permit release; the manager
does not impose a timeout or preempt it.
Component rewards and consumer support
reward.aggregation defaults to weighted_sum. Component mode requires
algorithm.trainer_type=direct_preference and the registered omni_nft loss.
The reward manager checks both at initialization, before reward workers start.
The OmniNFT loss consumes the named components and applies modality routing.
Online DPO, FlowGRPO, and ordinary DiffusionNFT do not support this layout.
For a supporting consumer, component mode returns rm_scores with shape [B, K].
Columns follow the sorted reward-function names in meta_info["reward_names"].
Every term must have required: true and weight: 1.0; missing or non-finite
components are rejected before actor update. Per-term reward/<name> metrics
remain available. Summing the columns produces an unweighted diagnostic total,
not a modality-weighted training objective. routing_weights is reserved for
the companion OmniNFT consumer and is not forwarded to scorers.
Use engine only
This example serves one model with two-way tensor parallelism:
reward:
reward_model:
enable: false
enable_resource_pool: true
n_gpus_per_node: 2
nnodes: 1
models:
ocr:
backend: engine
offload: true
model_path: Qwen/Qwen3-VL-8B-Instruct
n_gpus_per_node: 2
nnodes: 1
rollout:
name: vllm
tensor_model_parallel_size: 2
data_parallel_size: 1
pipeline_model_parallel_size: 1
reward_functions:
ocr:
path: pkg://verl_omni.utils.reward_score.genrm_ocr
name: compute_score_ocr
weight: 1.0
The engine owns serving, request batching, and TP/DP/PP. The reward function owns the request format and score calculation.
Use native only
This example starts one complete Transformers model replica on each of two native reward workers:
reward:
reward_model:
enable: false
enable_resource_pool: true
n_gpus_per_node: 2
nnodes: 1
models:
quality:
backend: native
offload: true
model_path: /models/quality
placement:
devices: [0, 1]
executor:
model: my_package.reward_model:TransformersRewardModel
kwargs:
torch_dtype: bfloat16
reward_functions:
quality:
path: pkg://my_package.reward_score
name: compute_quality_score
weight: 1.0
executor.model accepts an importable module:Class, a
pkg://module:Class, or a supported Python file path plus class name. The
framework supplies model_path and the worker-local device unless those
arguments are already present in executor.kwargs.
Each native model entry is one deployment. Different checkpoints or lifecycle
policies require different named deployments; multiple reward functions may
share one deployment through their model field. For accelerator placement, every
placement.devices entry creates one complete replica of that deployment.
Experimental FSDP2 PickScore support uses placement.replica_groups: [[0, 1]]
for one sharded replica across two GPUs. It requires a dedicated single-node
reward pool and synchronous V1 diffusion training. See the
FSDP2 reward model design for configuration,
lifecycle guarantees and the supported scope.
For native CLAP, AudioBox, and DeSync scorers, a single-sample batch supplies
audio, audio_sample_rate, and optional audio_layout in preference to
extra_info. Without batch, the scorers read extra_info.
AudioBox and DeSync require an explicit positive integer source sample rate.
Mono (T,) audio is accepted directly; two-dimensional audio requires
audio_layout: CT in the scorer config or metadata and is interpreted as
(channels, samples). Channels-last TC, empty waveforms, and non-finite
samples are rejected.
CLAP preserves its existing defaults: an absent sample-rate key means 48 kHz,
and two-dimensional audio is interpreted as (channels, samples) without a
new config field. An explicit source sample rate still takes precedence, so
24 kHz LTX and 32 kHz MiniMax audio are resampled from their actual rates.
Existing CLAP recipes do not need an audio_layout setting. CLAP shares the
empty/non-finite input checks, and accepts an explicit layout when supplied.
HPSv3 and VideoAlign accept uint8 frames or finite floating-point frames in
[0, 1]; values outside that range are rejected. HPSv3 applies the same pixel
validation to interval and uniform sampling.
Configure native audio/video scorers
Use pkg://verl_omni.utils.reward_score.<module> paths for the new AudioBox,
DeSync, and VideoAlign scorers. Bind a native model through model: (or a
matching model/term name) with executor.model set to module:Class:
audiobox:AudioBoxModel, desync:DeSyncModel, or videoalign:VideoAlignModel,
using the full verl_omni.utils.reward_score module prefix. DeSync additionally
requires source_root in executor.kwargs; VideoAlign requires base_model_path.
DeSync and VideoAlign require fps in each reward-function config. Set it to the
actual generated frame rate, including fractional rates; there is no inferred
or checkpoint-default FPS. For scorers that require a native reward_model,
missing required arguments are configuration errors and fail before optional
reward-error handling, even with required: false. If such a scorer also requires
fps, the manager rejects null, boolean, non-numeric, non-finite, zero, and negative
values before calling it. Positive fractional rates and numeric strings remain
supported. Optional model inference failures still contribute zero and record an
error; existing scorers retain their optional failure behavior.
For example, with a configured native desync model:
reward_functions:
desync:
path: pkg://verl_omni.utils.reward_score.desync
name: compute_score
model: desync
fps: 24.0 # Must match the generated video.
audio_layout: CT
required: true
VideoAlign requires a non-empty prompt from ground_truth or prompt_key.
DeSync currently requires uint8 RGB [T, C, H, W] video. HPSv3 and VideoAlign
also accept finite floating-point pixels in [0, 1]; this does not imply support
for arbitrary layouts. Normalize decoded video to TCHW at the caller boundary.
DeSync loads Python code from source_root, so use a trusted checkout. The
reference OmniNFT revision is fb9237f6e74edf0d0f2a683f4d975b79fde588fe; record
local modifications separately. Path checks do not authenticate source contents.
The existing loader and AST compatibility lookup use the upstream package
namespace; replacing only the entry-file import would not isolate that package.
Use native on CPU
Small native reward executors can run as ordinary Ray CPU actors without
allocating a trainer GPU pool. Set placement.resource=cpu, use
num_replicas for the number of complete replicas, and reserve CPU capacity
for each replica with cpus_per_worker:
reward:
models:
quality:
backend: native
model_path: /models/quality
placement:
resource: cpu
num_replicas: 2
cpus_per_worker: 2
executor:
model: my_package.reward_model:CpuRewardModel
reward_functions:
quality:
path: pkg://my_package.reward_score
name: compute_quality_score
CPU native executors default to torch.device("cpu"). An explicit
executor.kwargs.device must use CPU for CPU placement, or the current
accelerator type for accelerator placement. A mismatch is rejected before
workers are created. placement.devices is reserved for accelerator bundle
indices; CPU placement requires a positive num_replicas. CPU deployments do not consume or split
the trainer-selected accelerator reward pool, so they can be used alone or
alongside engine and accelerator-native deployments.
Each scoring phase wakes its worker-local models and, when offload=true,
unloads them after all accepted scoring calls finish. A scoring failure or
caller cancellation also waits for accepted calls before unloading. This wait
has no timeout: a scoring RPC that never returns can block cleanup. CPU
placement does not add forced worker termination or automatic retries.
With offload=false, models remain resident. If a CPU deployment explicitly enables
Ray actor restarts, the next scoring phase initializes a restarted actor again.
Production worker creation does not enable automatic restarts. Accelerator
deployments with offload=false skip repeated wake-ups and do not provide this
restart recovery.
Wrap a Transformers model for native mode
A Transformers checkpoint does not need an inference server. Add a small model
adapter that loads the processor and model and exposes infer(). The adapter
owns inference only; it must not decide the final reward semantics.
import torch
from transformers import AutoModel, AutoProcessor
class TransformersRewardModel:
def __init__(
self,
model_path: str,
device: torch.device,
torch_dtype: str = "bfloat16",
):
self.device = device
dtype = getattr(torch, torch_dtype)
self.processor = AutoProcessor.from_pretrained(model_path)
self.model = AutoModel.from_pretrained(
model_path,
torch_dtype=dtype,
).eval().to(device)
@torch.inference_mode()
def infer(self, texts, images):
inputs = self.processor(
text=texts,
images=images,
padding=True,
return_tensors="pt",
).to(self.device)
outputs = self.model(**inputs)
return {"logits": outputs.logits_per_image.detach().cpu()}
def close(self):
del self.model
del self.processor
Then add a separate score adapter. Its reward_model argument is the native
inference handle exposed by the framework:
async def compute_quality_score(
data_source,
solution_image,
ground_truth,
extra_info,
reward_model,
):
del data_source, extra_info
output = await reward_model.infer(
texts=[ground_truth or ""],
images=[solution_image],
)
return {"score": float(output["logits"][0, 0])}
The names and shapes passed to infer() are an internal contract between these
two adapters; the framework does not prescribe them. Both synchronous and
asynchronous infer() and close() implementations are accepted. Synchronous
methods run outside the reward worker’s event loop. close() is optional; the
executor also runs garbage collection and clears the accelerator cache when a
model sleeps.
Mix engine and native models
Engine and native models can be scored in the same job. The following dedicated four-device parent pool is split into a two-device engine allocation and an independent two-device native subpool:
reward:
reward_model:
enable: false
enable_resource_pool: true
n_gpus_per_node: 4
nnodes: 1
models:
ocr:
backend: engine
offload: true
model_path: /models/ocr
n_gpus_per_node: 2
nnodes: 1
rollout:
name: vllm
tensor_model_parallel_size: 2
quality:
backend: native
offload: true
model_path: /models/quality
placement:
devices: [2, 3]
executor:
model: my_package.reward_model:TransformersRewardModel
reward_functions:
ocr:
path: pkg://my_package.ocr_reward
name: compute_score
weight: 0.4
quality:
path: pkg://my_package.reward_score
name: compute_quality_score
weight: 0.6
All engine allocations are carved out first in configuration order. Each
native model then receives an independent subpool at the parent-pool bundle
indices listed in placement.devices. These are global indices within the
trainer-selected parent resource pool, not physical CUDA/NPU IDs and not
tensor-parallel ranks. Indices may be non-contiguous, but cannot overlap another
native model or the engine allocation. Each index creates one complete native
replica in this PR; it does not identify a rank in a sharded model group.
An engine model’s world size is
replicas * TP * DP * PP. If n_gpus_per_node and nnodes are set on that
model, their product must be at least the world size and a multiple of it. All
named allocations together must fit in the selected parent pool.
Set reward.reward_model.enable_resource_pool=false to split the trainer’s
global parent pool instead. Set it to true to create the dedicated parent pool
whose size is controlled by reward.reward_model.n_gpus_per_node and nnodes.
Lifecycle and execution
offload has the same meaning for both backends:
true(default): wake before scoring and sleep afterward;false: keep the model resident across training steps.
Independent named models are woken, scored, and slept concurrently. Native
batches are padded and split evenly across the workers assigned to that model
by default. Native deployments can opt into completion-driven microbatch dispatch
with dispatch_batch_size; see Native replica scheduling.
The reward loop exposes async_compute_rm_score() for asynchronous callers and
keeps compute_rm_score() as the synchronous compatibility entrypoint used by
current trainers. Cleanup is attempted even when inference or scoring fails.
Native PickScore can retain its existing weights on CPU while sleeping by setting
executor.kwargs.retain_weights_on_cpu: true. This opt-in requires a CUDA or Intel GPU worker;
the default native lifecycle still closes and reconstructs the model. CPU
retention keeps the same model and processor, and inference is rejected while
asleep. Call await multi_reward_model_manager.close_native_models() for final
native teardown when the caller owns the manager. This does not close engine
models. No automatic trainer teardown hook is provided for this opt-in.
PickScore validation recipe
The standard Qwen-Image-Edit launcher uses native PickScore. A mixed vLLM and
native parity recipe is available at
tests/special_e2e/run_qwen_image_edit_lora_v1_npu_engine_native.sh. It runs the
same reward through both backends for validation and is not a production
example.
Engine PickScore uses vLLM’s pooling runner and /v1/embeddings. Its reward
function computes:
PickScore = logit_scale * cosine(text_embedding, image_embedding) / 26
The configured logit_scale is already exponentiated and must not be passed
through exp() again.
Replica scheduling benchmark
The opt-in two-GPU benchmark compares static splitting with dynamic microbatches of 4 and 16 on the same pretrained PickScore replicas:
python tests/reward_loop/benchmark_replica_dispatch.py \
--model-path /path/to/PickScore_v1 \
--processor-path /path/to/clip_processor \
--output /path/to/benchmark-results.json
It warms each configuration and records six paired rounds of 128 samples,
alternating execution order and the slow replica. The synthetic delay is one
coefficient * len(sample_ids) sleep per RPC before that RPC queues concurrent
single-sample scoring; it occupies that replica’s sole in-flight RPC and is not
an independent per-sample scoring delay. It demonstrates conditional load
redistribution under this imposed imbalance, not a general scoring speedup or
natural model latency. Every arm checks score parity, ordering, exact-once
dispatch and bounded concurrency.
Timings cover dispatch and scoring with resident models and actor-cached images. They exclude model loading, offloading and full image-payload transfer, and are not end-to-end training measurements. Small microbatches can reduce batching efficiency: with one inference per sample, a size of 4 underfills PickScore’s 16-request consumer cap. Retain the static default unless a representative workload benefits.
Current limitations
Named-model aggregation currently uses the visual input contract implemented by
MultiVisualRewardManager; the aggregation core itself is modality-neutral.Flat native placement creates full replicas. Experimental FSDP2 support is limited to one frozen PickScore deployment with two GPU ranks; see the design document. Tensor parallelism is not supported.
Native routing defaults to a static even split; optional microbatch dispatch balances available work, without preempting or stealing an active batch.
Named models do not participate in streaming reward computation.
vLLM-Omni reward serving is not implemented.
Automatic migration of every existing reward implementation and a unified streaming integration remain follow-up work. The configuration migration and extension contracts supported by this change are documented above.