Reward Interface
Last updated: Oct 21, 2026 (API docstrings are auto-generated).
VeRL-Omni reward pipelines support both rule-based scoring (e.g. JPEG
compressibility) and model-based generative reward models (e.g. OCR via a
vision-language model served behind an OpenAI-compatible router). Reward
computation is dispatched per sample by reward managers. The input-agnostic
MultiRewardManager provides the
shared named-term dispatch, per-term outputs, failure handling, and weighted
aggregation used by input-specific subclasses. Modality-specific managers include
VisualRewardManager and
AudioRewardManager. They plug
into OmniRewardLoopManager — verl’s
RewardLoopManager extended with
profiler control over the reward-model rollout servers.
Compute the reward score for a visual (image) response. |
|
|
Compute reward by calling an external HTTP scorer service. |
|
Send one waveform to an external scorer and return its finite score. |
|
Compute a human-preference score via UnifiedReward 2.0. |
Reward Loop Manager
Reward Manager
AudioRewardManager reads audio and audio_sample_rate from rollout
extra_info, validates a finite CPU float waveform, and calls a synchronous
or asynchronous custom scorer with solution_audio=(waveform, sample_rate).
Default Score Dispatcher
Visual (image) reward scoring functions for VeRL-Omni.
- verl_omni.utils.reward_score.default_compute_score_image(data_source, solution_image, ground_truth, extra_info=None, **kwargs)[source]
Compute the reward score for a visual (image) response.
- Parameters:
data_source (str) – Dataset identifier that determines the scoring method.
solution_image – The generated image, as a uint8
torch.Tensorin[0, 255]with shape(C, H, W)or(N, C, H, W).ground_truth (str) – Ground-truth answer (may be unused for rule-based rewards such as
jpeg_compressibility).extra_info (dict, optional) – Additional metadata passed by the reward manager.
- Returns:
The computed score (or a dict with a
"score"key).- Return type:
float or dict
- Raises:
NotImplementedError – If no scorer is registered for data_source.
Built-in Reward Scorers
JPEG Compressibility
The reward function for JPEG compressibility. It is adapted from https://github.com/kvablack/ddpo-pytorch.
- verl_omni.utils.reward_score.jpeg_compressibility.compute_score(solution_image, extra_info=None)[source]
The scoring function for JPEG compressibility.
- Parameters:
solution_image – the solution image or video, in shape (C, H, W) or (N, C, H, W).
extra_info – artifact declarations or explicit
media_kind/fpsmetadata.
- Returns:
The first frame’s score; video rewards do not average across frames.
GRM-based OCR Reward
OCR scoring backed by a generative reward model (GRM).
The compute_score_ocr() function sends a generated image to a vision
language model served behind an OpenAI-compatible router and uses the model’s
transcription, compared to a ground truth, to produce a score in [0, 1].
- async verl_omni.utils.reward_score.genrm_ocr.compute_score_ocr(data_source: str, solution_image: ndarray | Tensor, ground_truth: str, extra_info: dict, reward_router_address: str, reward_model_tokenizer: PythonBackend = None, model_name: str | None = None, sampling_params: dict = None)[source]
Compute an image OCR score via a generative reward model (GRM).
The image is sent to a GRM via an OpenAI-compatible router; the returned transcription is compared to
ground_truthusing Levenshtein distance to yield a score in[0, 1](1 = perfect match).- Parameters:
data_source – Source dataset identifier. Unused, kept for interface consistency.
solution_image – The solution image or video to be evaluated.
ground_truth – The ground truth text for comparison.
extra_info – Additional information;
frame_intervalcontrols video frame subsampling.reward_router_address –
host:portof the GRM router.reward_model_tokenizer – Tokenizer for the reward model. Unused, kept for interface consistency.
model_name – Name or path of the GRM. Defaults to
DEFAULT_GRM_MODEL_PATH.sampling_params – Override the defaults from
_sampling_params(). An explicitseedopts in to reproducible sampling; without one any env-inherited seed is dropped.
- Returns:
{"score": float, "genrm_response": str}.- Return type:
dict
HTTP Scorer Client
Generic HTTP reward client for external scorer services.
Sends generated images to an external HTTP scorer service using pickle protocol and returns the score. Compatible with all scorer services under rewards_services/api_services/ that accept the standard payload format:
POST with pickle-serialized {"images": List[bytes], "prompts": List[str], "metadata": dict}
Response: pickle-serialized {"scores": List[float]}
- async verl_omni.utils.reward_score.http_scorer_client.compute_score(solution_image: Tensor, ground_truth: str, server_url: str, max_retries: int = 2, retry_backoff: float = 0.5, **kwargs) dict[source]
Compute reward by calling an external HTTP scorer service.
- Parameters:
solution_image – Generated image tensor (C, H, W) or (N, C, H, W).
ground_truth – Prompt string passed directly to the scorer service.
server_url – Full URL of the scorer service (e.g., “http://localhost:19082”).
max_retries – Number of retries after the initial HTTP attempt.
retry_backoff – Initial delay in seconds between retries. The delay doubles after each failure.
- Returns:
dict with “score” key.
- Raises:
RuntimeError – If the scorer returns an HTTP error, reports an error, or returns no scores.
Audio HTTP Scorer Client
JSON HTTP client for audio reward services in a separate runtime.
- async verl_omni.utils.reward_score.audio_http_scorer_client.compute_score(solution_audio, ground_truth: str, extra_info: dict | None = None, data_source: str | None = None, *, server_url: str, timeout: float = 120.0, max_retries: int = 2, retry_backoff: float = 0.5) dict[source]
Send one waveform to an external scorer and return its finite score.
UnifiedReward Scorer
Human-preference scoring backed by UnifiedReward 2.0.
- async verl_omni.utils.reward_score.unified_reward.compute_score_unified_reward(data_source: str, solution_image: ndarray | Tensor, ground_truth: str, extra_info: dict, reward_router_address: str, reward_model_tokenizer: PythonBackend = None, model_name: str | None = None)[source]
Compute a human-preference score via UnifiedReward 2.0.
The reward model scores the generated image against its text caption on Alignment, Coherence, and Style axes. The returned
scoreis the mean of those axes normalized from the model’s 1-5 scale to[0, 1].
Reward Utilities
Image/video conversion helpers shared by reward scoring and trainer output paths.