Named Reward Models

Last updated: 11/13/2026

This guide describes how to configure and extend named model-backed rewards under reward.models in verl-omni. For the general Reward Loop interface and custom reward functions, refer to the upstream verl documentation.

reward.models lets one training job use one or more independently managed model-backed rewards. It supports an engine backend, a native backend, or both in the same job.

The framework deliberately separates inference from scoring:

  • a named model owns resources, inference access, and lifecycle;

  • a reward function converts one training sample into model inputs and converts the model output into a score;

  • MultiRewardManager owns model-aware scorer dispatch, per-term outputs, failure handling, and weighted aggregation without owning an input modality.

  • An input-specific subclass prepares scorer arguments. The current named-model recipes use the visual contract supplied by MultiVisualRewardManager.

PickScore is an example of this contract, not a special case in the framework.

Named models require a MultiRewardManager subclass with an input contract. Current maintained recipes use the visual subclass explicitly; the framework does not rewrite a user-provided manager:

reward:
  reward_manager:
    name: MultiVisualRewardManager

Consolidating audio, text, and other input contracts behind the shared manager is separate follow-up work. Until then, modality-specific managers and their existing recipes remain unchanged.

Native replica scheduling

Named models use static padded splitting by default: each worker receives one equal-sized chunk. For native models whose sample costs or replica speeds vary, set reward.models.<name>.dispatch_batch_size to a positive integer to enable completion-driven dispatch:

reward:
  models:
    pickscore:
      backend: native
      placement:
        devices: [0, 1]
      executor:
        model: verl_omni.utils.reward_score.pickscore_reward:PickScoreNativeModel
      dispatch_batch_size: 8

Keep the model’s executor arguments and reward function configured as described below. Each placement entry still owns one complete model replica. Each replica receives at most one scoring RPC at a time and takes the next contiguous batch when it finishes. The final batch may be smaller; no duplicate padding is added. Results are restored to input order before existing per-model score aggregation. Omit this field or set it to null to keep static splitting. Engine models reject this option; their internal scheduling and parallelism remain engine-owned.

Choose a batch size large enough for efficient model batching but small enough to leave work available for faster replicas. It caps the in-flight samples per replica, so it also limits how many single-sample requests can be waiting for a model consumer. For PickScore, whose native consumer caps each forward at 16, one inference per sample and dispatch_batch_size: 8 can coalesce at most eight requests. Values above 16 can reduce RPC count but cannot raise that forward batch cap; an actual forward can still be smaller. Opt in only when samples can be scored independently: batch-sensitive or replica-local random scorers can change scores when batch boundaries or replica assignments change.

On a dispatched scoring failure or caller cancellation, no new batches are submitted once the dispatcher observes it. Already submitted RPCs are drained before the error is raised and models sleep, including across mixed engine/native groups. There is no automatic retry, actor recovery, speculative execution, autoscaling or streaming-trainer support. A stuck RPC can therefore delay drain; this option does not introduce a timeout or health-check policy.

Backend selection

Backend

Use it when

Reward-function arguments

engine

The model is supported by the existing verl.RewardModelManager and should be served by vLLM

reward_router_address and model_name

native

The model can be loaded and called directly in a reward worker, including ordinary Transformers models

reward_model, an inference handle

The engine path documented here uses vLLM. vLLM-Omni reward serving is not implemented; vLLM-Omni can still be used independently for actor rollout.

Existing jobs without reward.models continue to use the existing single-model reward path. Do not set reward.reward_model.enable=true together with reward.models.

Migrate an existing engine reward

An existing single-model engine configuration has one global reward model and one custom reward function:

reward:
  reward_model:
    enable: true
    enable_resource_pool: true
    n_gpus_per_node: 2
    nnodes: 1
    model_path: /models/ocr
    rollout:
      name: vllm
      tensor_model_parallel_size: 2

  custom_reward_function:
    path: pkg://my_package.ocr_reward
    name: compute_score

To migrate it, disable the existing single model, create a named engine model, and move the scoring function into reward.reward_functions. The model name and reward term name can be the same:

reward:
  reward_model:
    enable: false
    enable_resource_pool: true
    n_gpus_per_node: 2
    nnodes: 1

  models:
    ocr:
      backend: engine
      model_path: /models/ocr
      n_gpus_per_node: 2
      nnodes: 1
      rollout:
        name: vllm
        tensor_model_parallel_size: 2

  reward_functions:
    ocr:
      path: pkg://my_package.ocr_reward
      name: compute_score
      weight: 1.0
      required: true

Remove the old custom_reward_function.path override when switching to reward_functions. The score function can keep the existing router-based contract if it already accepts reward_router_address and model_name:

async def compute_score(
    data_source,
    solution_image,
    ground_truth,
    extra_info,
    reward_router_address,
    model_name,
):
    response = await call_openai_compatible_server(
        address=reward_router_address,
        model=model_name,
        image=solution_image,
        prompt=ground_truth,
    )
    return {"score": parse_score(response)}

reward.reward_model.rollout remains the common engine default. Values under a named model’s rollout override those defaults. A named model’s model_path also overrides the common reward_model.model_path fallback.

SD3.5 V1 OCR recipe

The SD3.5 V1 synchronous recipe uses reward.models.ocr with the engine backend and MultiVisualRewardManager. It keeps the existing Qwen/Qwen2.5-VL-3B-Instruct checkpoint and compute_score_ocr scorer, with weight 1.0 and required=true.

Prepare data/ocr/sd3/train.parquet and data/ocr/sd3/test.parquet under OCR_WORKSPACE, then run from the repository root:

OCR_WORKSPACE=/path/to/workspace bash examples/flowgrpo_trainer/sd35/run_sd35_medium_ocr_lora_v1.sh \
  'trainer.logger=[console]'

Caller overrides remain last. This recipe opts into reward.reward_functions.ocr.use_rollout_sampling_params=true to forward the response length and optional deterministic seed from the model’s rollout settings, falling back to reward.reward_model.rollout. Explicit scorer sampling_params take precedence. Existing named rewards without this opt-in keep their scorer defaults; OCR still defaults to 4096 output tokens and honors GENRM_OCR_SEED. The opt-in preserves the legacy visual manager’s seed behavior: an environment seed is excluded unless rollout determinism supplies a seed.

Model-to-reward binding

A reward term automatically uses a model with the same name:

reward:
  models:
    quality:
      backend: native
      model_path: /models/quality
      placement:
        devices: [0]
      executor:
        model: my_package.reward_model:QualityModel

  reward_functions:
    quality:
      path: pkg://my_package.reward_score
      name: compute_quality_score

Set model explicitly when the names differ or several reward terms share one model:

reward_functions:
  semantic_quality:
    model: quality
    path: pkg://my_package.reward_score
    name: compute_semantic_quality

VLM-as-a-judge

One judge model can serve several reward terms, each with its own user_prompt (literal text, or a pkg:// builder). For UnifiedReward, literal text may contain {caption}, and a custom builder receives the caption. Leaving user_prompt unset keeps the scorer’s default request.

reward_functions:
  ocr:
    model: judge
    path: pkg://verl_omni.utils.reward_score.genrm_ocr
    name: compute_score_ocr
    user_prompt: "Transcribe all text in the image exactly."
  semantics:
    model: judge
    path: pkg://verl_omni.utils.reward_score.unified_reward
    name: compute_score_unified_reward
    user_prompt: pkg://my_package.prompts.build_semantic_prompt

Every term uses the existing aggregation contract:

final_reward = sum(term.weight * term.score)

required=true makes a scoring failure fatal. An optional term records the error and contributes zero. Model setup and lifecycle failures are always fatal. ArtifactContractError is fatal for every term, including optional terms, so missing or invalid named media cannot silently contribute zero.

Concurrent independent reward terms

By default, reward terms run in configuration order within each sample, while separate samples retain their existing concurrency. To overlap independent terms, set a worker-wide cap above one and opt in each term:

reward:
  multi_reward_concurrency: 2
  reward_functions:
    first:
      path: pkg://my_package.rewards
      name: score_first
      independent: true
    second:
      path: pkg://my_package.rewards
      name: score_second
      independent: true

independent must be a boolean and defaults to false. Eligible terms use coroutine score functions with no named model (for example, an external HTTP scorer) or an engine named model. Native model terms and synchronous score functions cannot opt in. Opting in asserts that score functions treat sample inputs as read-only, do not share mutable state, and have no dependencies on other reward terms. The manager passes the same input objects to each scorer; it does not copy large image or other payloads.

Adjacent independent terms execute in windows of at most multi_reward_concurrency; any other term is an ordering barrier for its sample. When the cap is above one, it bounds active independent scorer calls across all samples handled by one reward worker. Barriers do not use the cap and retain cross-sample concurrency, even while independent calls are active. Each window waits for all its terms to settle before admitting the next window for that sample, even if one term finishes and frees a semaphore permit while another is still running. This bounds the number of admitted tasks rather than enqueueing every term at once; a freed permit remains available to independent terms from other samples. Scores, diagnostics, and required/optional error handling remain in configuration order. The cap does not limit reward workers, barrier calls, or engine-side requests. Increasing it does not establish a throughput improvement by itself. Cancelling a serial synchronous scorer stops its caller without waiting for its executor thread; the thread may continue until the scorer returns. An admitted window settles all its terms before reporting a contract error. Independent async scorers must cooperate with cancellation. A scorer that swallows CancelledError can block window drain and permit release; the manager does not impose a timeout or preempt it.

Component rewards and consumer support

reward.aggregation defaults to weighted_sum. Component mode requires algorithm.trainer_type=direct_preference and the registered omni_nft loss. The reward manager checks both at initialization, before reward workers start. The OmniNFT loss consumes the named components and applies modality routing. Online DPO, FlowGRPO, and ordinary DiffusionNFT do not support this layout.

For a supporting consumer, component mode returns rm_scores with shape [B, K]. Columns follow the sorted reward-function names in meta_info["reward_names"]. Every term must have required: true and weight: 1.0; missing or non-finite components are rejected before actor update. Per-term reward/<name> metrics remain available. Summing the columns produces an unweighted diagnostic total, not a modality-weighted training objective. routing_weights is reserved for the companion OmniNFT consumer and is not forwarded to scorers.

Use engine only

This example serves one model with two-way tensor parallelism:

reward:
  reward_model:
    enable: false
    enable_resource_pool: true
    n_gpus_per_node: 2
    nnodes: 1

  models:
    ocr:
      backend: engine
      offload: true
      model_path: Qwen/Qwen3-VL-8B-Instruct
      n_gpus_per_node: 2
      nnodes: 1
      rollout:
        name: vllm
        tensor_model_parallel_size: 2
        data_parallel_size: 1
        pipeline_model_parallel_size: 1

  reward_functions:
    ocr:
      path: pkg://verl_omni.utils.reward_score.genrm_ocr
      name: compute_score_ocr
      weight: 1.0

The engine owns serving, request batching, and TP/DP/PP. The reward function owns the request format and score calculation.

Use native only

This example starts one complete Transformers model replica on each of two native reward workers:

reward:
  reward_model:
    enable: false
    enable_resource_pool: true
    n_gpus_per_node: 2
    nnodes: 1

  models:
    quality:
      backend: native
      offload: true
      model_path: /models/quality
      placement:
        devices: [0, 1]
      executor:
        model: my_package.reward_model:TransformersRewardModel
        kwargs:
          torch_dtype: bfloat16

  reward_functions:
    quality:
      path: pkg://my_package.reward_score
      name: compute_quality_score
      weight: 1.0

executor.model accepts an importable module:Class, a pkg://module:Class, or a supported Python file path plus class name. The framework supplies model_path and the worker-local device unless those arguments are already present in executor.kwargs.

Each native model entry is one deployment. Different checkpoints or lifecycle policies require different named deployments; multiple reward functions may share one deployment through their model field. For accelerator placement, every placement.devices entry creates one complete replica of that deployment. Experimental FSDP2 PickScore support uses placement.replica_groups: [[0, 1]] for one sharded replica across two GPUs. It requires a dedicated single-node reward pool and synchronous V1 diffusion training. See the FSDP2 reward model design for configuration, lifecycle guarantees and the supported scope.

For native CLAP, AudioBox, and DeSync scorers, a single-sample batch supplies audio, audio_sample_rate, and optional audio_layout in preference to extra_info. Without batch, the scorers read extra_info.

AudioBox and DeSync require an explicit positive integer source sample rate. Mono (T,) audio is accepted directly; two-dimensional audio requires audio_layout: CT in the scorer config or metadata and is interpreted as (channels, samples). Channels-last TC, empty waveforms, and non-finite samples are rejected.

CLAP preserves its existing defaults: an absent sample-rate key means 48 kHz, and two-dimensional audio is interpreted as (channels, samples) without a new config field. An explicit source sample rate still takes precedence, so 24 kHz LTX and 32 kHz MiniMax audio are resampled from their actual rates. Existing CLAP recipes do not need an audio_layout setting. CLAP shares the empty/non-finite input checks, and accepts an explicit layout when supplied.

HPSv3 and VideoAlign accept uint8 frames or finite floating-point frames in [0, 1]; values outside that range are rejected. HPSv3 applies the same pixel validation to interval and uniform sampling.

Configure native audio/video scorers

Use pkg://verl_omni.utils.reward_score.<module> paths for the new AudioBox, DeSync, and VideoAlign scorers. Bind a native model through model: (or a matching model/term name) with executor.model set to module:Class: audiobox:AudioBoxModel, desync:DeSyncModel, or videoalign:VideoAlignModel, using the full verl_omni.utils.reward_score module prefix. DeSync additionally requires source_root in executor.kwargs; VideoAlign requires base_model_path.

DeSync and VideoAlign require fps in each reward-function config. Set it to the actual generated frame rate, including fractional rates; there is no inferred or checkpoint-default FPS. For scorers that require a native reward_model, missing required arguments are configuration errors and fail before optional reward-error handling, even with required: false. If such a scorer also requires fps, the manager rejects null, boolean, non-numeric, non-finite, zero, and negative values before calling it. Positive fractional rates and numeric strings remain supported. Optional model inference failures still contribute zero and record an error; existing scorers retain their optional failure behavior. For example, with a configured native desync model:

reward_functions:
  desync:
    path: pkg://verl_omni.utils.reward_score.desync
    name: compute_score
    model: desync
    fps: 24.0  # Must match the generated video.
    audio_layout: CT
    required: true

VideoAlign requires a non-empty prompt from ground_truth or prompt_key. DeSync currently requires uint8 RGB [T, C, H, W] video. HPSv3 and VideoAlign also accept finite floating-point pixels in [0, 1]; this does not imply support for arbitrary layouts. Normalize decoded video to TCHW at the caller boundary.

DeSync loads Python code from source_root, so use a trusted checkout. The reference OmniNFT revision is fb9237f6e74edf0d0f2a683f4d975b79fde588fe; record local modifications separately. Path checks do not authenticate source contents. The existing loader and AST compatibility lookup use the upstream package namespace; replacing only the entry-file import would not isolate that package.

Use native on CPU

Small native reward executors can run as ordinary Ray CPU actors without allocating a trainer GPU pool. Set placement.resource=cpu, use num_replicas for the number of complete replicas, and reserve CPU capacity for each replica with cpus_per_worker:

reward:
  models:
    quality:
      backend: native
      model_path: /models/quality
      placement:
        resource: cpu
        num_replicas: 2
        cpus_per_worker: 2
      executor:
        model: my_package.reward_model:CpuRewardModel
  reward_functions:
    quality:
      path: pkg://my_package.reward_score
      name: compute_quality_score

CPU native executors default to torch.device("cpu"). An explicit executor.kwargs.device must use CPU for CPU placement, or the current accelerator type for accelerator placement. A mismatch is rejected before workers are created. placement.devices is reserved for accelerator bundle indices; CPU placement requires a positive num_replicas. CPU deployments do not consume or split the trainer-selected accelerator reward pool, so they can be used alone or alongside engine and accelerator-native deployments.

Each scoring phase wakes its worker-local models and, when offload=true, unloads them after all accepted scoring calls finish. A scoring failure or caller cancellation also waits for accepted calls before unloading. This wait has no timeout: a scoring RPC that never returns can block cleanup. CPU placement does not add forced worker termination or automatic retries.

With offload=false, models remain resident. If a CPU deployment explicitly enables Ray actor restarts, the next scoring phase initializes a restarted actor again. Production worker creation does not enable automatic restarts. Accelerator deployments with offload=false skip repeated wake-ups and do not provide this restart recovery.

Wrap a Transformers model for native mode

A Transformers checkpoint does not need an inference server. Add a small model adapter that loads the processor and model and exposes infer(). The adapter owns inference only; it must not decide the final reward semantics.

import torch
from transformers import AutoModel, AutoProcessor


class TransformersRewardModel:
    def __init__(
        self,
        model_path: str,
        device: torch.device,
        torch_dtype: str = "bfloat16",
    ):
        self.device = device
        dtype = getattr(torch, torch_dtype)
        self.processor = AutoProcessor.from_pretrained(model_path)
        self.model = AutoModel.from_pretrained(
            model_path,
            torch_dtype=dtype,
        ).eval().to(device)

    @torch.inference_mode()
    def infer(self, texts, images):
        inputs = self.processor(
            text=texts,
            images=images,
            padding=True,
            return_tensors="pt",
        ).to(self.device)
        outputs = self.model(**inputs)
        return {"logits": outputs.logits_per_image.detach().cpu()}

    def close(self):
        del self.model
        del self.processor

Then add a separate score adapter. Its reward_model argument is the native inference handle exposed by the framework:

async def compute_quality_score(
    data_source,
    solution_image,
    ground_truth,
    extra_info,
    reward_model,
):
    del data_source, extra_info
    output = await reward_model.infer(
        texts=[ground_truth or ""],
        images=[solution_image],
    )
    return {"score": float(output["logits"][0, 0])}

The names and shapes passed to infer() are an internal contract between these two adapters; the framework does not prescribe them. Both synchronous and asynchronous infer() and close() implementations are accepted. Synchronous methods run outside the reward worker’s event loop. close() is optional; the executor also runs garbage collection and clears the accelerator cache when a model sleeps.

Mix engine and native models

Engine and native models can be scored in the same job. The following dedicated four-device parent pool is split into a two-device engine allocation and an independent two-device native subpool:

reward:
  reward_model:
    enable: false
    enable_resource_pool: true
    n_gpus_per_node: 4
    nnodes: 1

  models:
    ocr:
      backend: engine
      offload: true
      model_path: /models/ocr
      n_gpus_per_node: 2
      nnodes: 1
      rollout:
        name: vllm
        tensor_model_parallel_size: 2

    quality:
      backend: native
      offload: true
      model_path: /models/quality
      placement:
        devices: [2, 3]
      executor:
        model: my_package.reward_model:TransformersRewardModel

  reward_functions:
    ocr:
      path: pkg://my_package.ocr_reward
      name: compute_score
      weight: 0.4
    quality:
      path: pkg://my_package.reward_score
      name: compute_quality_score
      weight: 0.6

All engine allocations are carved out first in configuration order. Each native model then receives an independent subpool at the parent-pool bundle indices listed in placement.devices. These are global indices within the trainer-selected parent resource pool, not physical CUDA/NPU IDs and not tensor-parallel ranks. Indices may be non-contiguous, but cannot overlap another native model or the engine allocation. Each index creates one complete native replica in this PR; it does not identify a rank in a sharded model group.

An engine model’s world size is replicas * TP * DP * PP. If n_gpus_per_node and nnodes are set on that model, their product must be at least the world size and a multiple of it. All named allocations together must fit in the selected parent pool.

Set reward.reward_model.enable_resource_pool=false to split the trainer’s global parent pool instead. Set it to true to create the dedicated parent pool whose size is controlled by reward.reward_model.n_gpus_per_node and nnodes.

Lifecycle and execution

offload has the same meaning for both backends:

  • true (default): wake before scoring and sleep afterward;

  • false: keep the model resident across training steps.

Independent named models are woken, scored, and slept concurrently. Native batches are padded and split evenly across the workers assigned to that model by default. Native deployments can opt into completion-driven microbatch dispatch with dispatch_batch_size; see Native replica scheduling.

The reward loop exposes async_compute_rm_score() for asynchronous callers and keeps compute_rm_score() as the synchronous compatibility entrypoint used by current trainers. Cleanup is attempted even when inference or scoring fails.

Native PickScore can retain its existing weights on CPU while sleeping by setting executor.kwargs.retain_weights_on_cpu: true. This opt-in requires a CUDA or Intel GPU worker; the default native lifecycle still closes and reconstructs the model. CPU retention keeps the same model and processor, and inference is rejected while asleep. Call await multi_reward_model_manager.close_native_models() for final native teardown when the caller owns the manager. This does not close engine models. No automatic trainer teardown hook is provided for this opt-in.

PickScore validation recipe

The standard Qwen-Image-Edit launcher uses native PickScore. A mixed vLLM and native parity recipe is available at tests/special_e2e/run_qwen_image_edit_lora_v1_npu_engine_native.sh. It runs the same reward through both backends for validation and is not a production example.

Engine PickScore uses vLLM’s pooling runner and /v1/embeddings. Its reward function computes:

PickScore = logit_scale * cosine(text_embedding, image_embedding) / 26

The configured logit_scale is already exponentiated and must not be passed through exp() again.

Replica scheduling benchmark

The opt-in two-GPU benchmark compares static splitting with dynamic microbatches of 4 and 16 on the same pretrained PickScore replicas:

python tests/reward_loop/benchmark_replica_dispatch.py \
  --model-path /path/to/PickScore_v1 \
  --processor-path /path/to/clip_processor \
  --output /path/to/benchmark-results.json

It warms each configuration and records six paired rounds of 128 samples, alternating execution order and the slow replica. The synthetic delay is one coefficient * len(sample_ids) sleep per RPC before that RPC queues concurrent single-sample scoring; it occupies that replica’s sole in-flight RPC and is not an independent per-sample scoring delay. It demonstrates conditional load redistribution under this imposed imbalance, not a general scoring speedup or natural model latency. Every arm checks score parity, ordering, exact-once dispatch and bounded concurrency.

Timings cover dispatch and scoring with resident models and actor-cached images. They exclude model loading, offloading and full image-payload transfer, and are not end-to-end training measurements. Small microbatches can reduce batching efficiency: with one inference per sample, a size of 4 underfills PickScore’s 16-request consumer cap. Retain the static default unless a representative workload benefits.

Current limitations

  • Named-model aggregation currently uses the visual input contract implemented by MultiVisualRewardManager; the aggregation core itself is modality-neutral.

  • Flat native placement creates full replicas. Experimental FSDP2 support is limited to one frozen PickScore deployment with two GPU ranks; see the design document. Tensor parallelism is not supported.

  • Native routing defaults to a static even split; optional microbatch dispatch balances available work, without preempting or stealing an active batch.

  • Named models do not participate in streaming reward computation.

  • vLLM-Omni reward serving is not implemented.

Automatic migration of every existing reward implementation and a unified streaming integration remain follow-up work. The configuration migration and extension contracts supported by this change are documented above.