Parallelize independent rubric judge calls #3040

Parent: #3001

Slowdown and pressure path

flowchart LR
    A[One verify with N rubric items] --> B{Caller scheduling}
    B --> C[Sequential independent calls]
    B --> D[gather N calls]
    C --> E[Longer verify latency]
    D --> F[Unbounded judge fanout]
    E --> G[Verifier completes]
    F --> G

Current behavior

At upstream main commit 9287fb7797907ff17884963730d79f428a225639, the main rubric callers already preserve input order with asyncio.gather when parallel evaluation is enabled:

• inverse_if gathers one _evaluate_criterion call per rubric item.
• multichallenge gathers one _evaluate_rubric_item call per item.
• The policy-aware path in jailbreak_detection also gathers its configured verifier calls.

These parallel paths currently have no shared concurrency bound because call_judge lacks one (#3023). In jailbreak_detection combined-reward mode, the safety and quality prompts are awaited sequentially (source). That path uses a bespoke request helper which catches infrastructure failures and returns an unclear score (source); failure classification must be fixed independently of scheduling.

The two math_with_judge order checks are not independent under current behavior: the second call is issued only when the first says equal, and the returned evaluation list can contain one or two entries (source). Running both concurrently would change cost and observable results, so it is outside this issue unless that contract is deliberately changed.

Impact and provenance

Measured

The 2026-08-31 report at investigations/gym-and-performance.md was recorded against 6992a96de. It did not measure rubric-judge latency or throughput. The report's high judge-concurrency figure is an extrapolation from step size and rubric fanout, not an observed peak. Current source also shows that the broad statement “rubric callers are sequential” is no longer accurate.

Expected

Expected, not yet measured: concurrent independent calls can reduce one verify's wall time toward the slowest item instead of the sum, but only when judge capacity is available. Unbounded gather can instead deepen the judge queue and worsen tail latency. No speedup should be claimed until #3023 provides a bound and the benchmark below is run.

Correctness and process behavior

Parallelization must preserve rubric-to-result ordering, aggregation, and failure attribution. Migrating jailbreak_detection from unclear-score fallback to shared JudgeError handling is a correctness change, not a throughput result. Conditional or short-circuit judge calls must remain sequential unless their externally visible semantics are intentionally revised.

Implementation constraints

• Land or use the shared semaphore and deadline from Add shared judge concurrency limits and deadlines #3023 before expanding fanout.
• Parallelize only calls whose prompts and scoring do not depend on another result.
• Preserve deterministic output order even when completion order differs.
• Define whole-group behavior for one timeout, one JudgeError, and caller cancellation; ensure sibling tasks are awaited or cancelled and drained.
• Do not retry judge calls implicitly.
• Keep parallel_evaluation: false behavior available where currently supported.
• Measure judge saturation and queue wait; lower verify latency must not be bought with uncontrolled model-server load.

Acceptance criteria

• The audit lists every changed caller and explains why its calls are independent.
• Existing gather callers are routed through the shared bound and deadline.
• Any newly parallel caller preserves call count, result order, aggregation, and evaluation-record shape.
• Conditional math_with_judge ordering remains unchanged.
• Tests cover mixed completion order, one timeout, one HTTP failure, malformed verdict, caller cancellation, and sequential mode.
• Sibling tasks do not leak after group failure or cancellation.
• jailbreak_detection failure classification is validated separately from latency.
• Benchmark results report both verifier latency and judge pressure.

Benchmark plan

1. Use a deterministic judge stub with configurable latency, jitter, timeout, and HTTP failure.
2. Run rubric sizes 1, 2, 4, 10, and 32 at verify concurrency 1, 32, 256, and 8,192.
3. Compare sequential mode, current unbounded gather, and bounded gather under identical offered load.
4. Record verify p50/p95/p99, completions/s, maximum resident judge calls, shared-semaphore wait, judge queue/service latency, cancellation count, failure classification, and result-order parity.
5. Repeat with a capacity-limited judge stub to identify when added fanout stops helping.
6. Report correctness differentials first; label latency and throughput as measured only from these runs.

Related work

• Parent resources-server workstream: Performance workstream: resources servers #3001
• Shared judge limits and deadlines: Add shared judge concurrency limits and deadlines #3023
• math_with_judge worker lifecycle: Replace math_with_judge fork polling with bounded warm execution #3025
• Unified judge failure handling: feat: unified judge failure handling #2113
• Umbrella performance issue: Performance opportunities across the NeMo Gym server stack #2998