Testing Guide

Last updated: 07/14/2026.

This guide explains the test hierarchy for verl_omni, starting with L1 CPU tests and leaving room for higher layers such as L2 GPU smoke tests.

Table of Contents

Test Hierarchy

The project uses a layered testing model. Lower layers should be cheaper, faster, and more deterministic. Choose the lowest layer that can catch the regression.

Layer

Purpose

Typical Environment

L1

CPU-only unit and lightweight integration tests

GitHub-hosted CPU runner

L2

GPU smoke tests for tiny end-to-end paths

GPU runner with tiny-random models

L3

Backend or numerical comparison tests

GPU runner with fixed seeds

L4

Real model and dataset validation

Scheduled or manually triggered jobs

Choosing a Layer

Question

Layer

Does it run fully on CPU and validate API, config, data, reward, adapter, or utility behavior?

L1

Does it require GPU and run a tiny-random model for one or two end-to-end training steps?

L2

Does it compare numerical metrics or performance across backends for short fixed-seed runs?

L3

Does it use real model weights and real datasets to validate convergence curves?

L4

L1 CPU Tests

L1 is the current primary CI test layer. Use it for lightweight CPU coverage that can run quickly and deterministically on every pull request.

Scope

Use L1 for lightweight coverage of:

  • Config defaults, validation, and Hydra composition.

  • Dataset parsing, tensor conversion, collation helpers, and data-source metadata.

  • Reward-score functions and reward managers that run without model inference.

  • Loss functions, advantage computation, registries, and metric utilities.

  • Pipeline or adapter boundary behavior that can be tested with mocks.

Do not use L1 for GPU kernels, Ray clusters, rollout engines, real checkpoints, or full trainer smoke scripts. Put those in L2 or above.

File Naming

L1 tests must end with _on_cpu.py. The CPU workflow writes a temporary pytest.ini that sets:

[pytest]
python_files = *_on_cpu.py

Placement

Place tests under the top-level module they cover. For example:

  • verl_omni/trainer/... -> tests/trainer/...

  • verl_omni/workers/... -> tests/workers/...

  • verl_omni/utils/... -> tests/utils/...

  • verl_omni/pipelines/... -> tests/pipelines/...

  • verl_omni/reward_loop/... -> tests/reward_loop/...

Special workflow folders such as tests/special_e2e/ and tests/special_sanity/ are reserved for non-L1 checks.

Coverage

L1 reports line and branch coverage for verl_omni. The workflow produces:

  • A terminal term-missing report in the job log.

  • coverage.xml for tooling and artifact download.

  • pytest-coverage.txt for the GitHub summary and artifact download.

Coverage should help identify untested API surfaces, but avoid adding brittle tests just to raise a number. Prefer tests that describe stable behavior and would catch a real regression.

Diff coverage thresholds should be introduced only after the baseline and exception policy are agreed by maintainers. Until then, contributors should use the report to inspect their changed modules.

Local Commands

To run only L1-style tests locally:

printf '[pytest]\npython_files = *_on_cpu.py\n' > pytest.ini
pytest -s -x --asyncio-mode=auto --cov=verl_omni --cov-report=term-missing --cov-report=xml:coverage.xml tests/

To run a single new L1 test file while iterating:

pytest -s -x --asyncio-mode=auto tests/path/to/test_file_on_cpu.py

Delete the temporary pytest.ini if it is not part of your intended change.

Adding a New L1 Test

  1. Confirm the behavior can run fully on CPU.

  2. Place the file under the matching tests/<module>/ directory.

  3. Name the file test_<behavior>_on_cpu.py.

  4. Keep fixtures small and deterministic.

  5. Mock model loading and external services.

  6. Run the L1 command above before opening a PR.

Future Layers

This guide currently defines L1 in detail because L1 is the main pull-request test layer. Add dedicated sections for L2, L3, or L4 when those workflows have stable ownership, trigger rules, naming conventions, and local commands.

When adding a new layer section, include:

  1. Scope and examples.

  2. Required environment.

  3. File naming or folder conventions.

  4. CI workflow trigger rules.

  5. Local or manual run commands.