Skip to content

fix(rollout): exclude compaction helpers from fallback turns - #5122

Merged
giuliolovisotto merged 3 commits into
mainfrom
codex/fea-518-collector
Nov 4, 2026
Merged

giuliolovisotto merged 3 commits into
mainfrom
codex/fea-518-collector

Conversation

@giuliolovisotto

@giuliolovisotto giuliolovisotto commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Automatic compaction produces a model call for the summary, but the collector currently turns that helper response into an extra policy turn. Exclude uniquely resolved compaction helper references from fallback turn construction while retaining the calls, ownership and token usage. Keep helpers out of turns even when agent ownership claims conflict, preserving those claims for validation. This matches the verifier exclusion rule in #4630. Preserve producer-provided trajectories and the HTTP-error exclusion introduced in #4045.

Stack: this PR → shared support #5307 → Kilo/Codex/Stirrup/Hermes fixes → #5439, which registers the scenario last.

Earlier validation:

  • 32 focused collector tests passed, including all reference forms, ownership conflicts in both invocation orders, ambiguous/mismatched references, retained usage and HTTP errors. Six new regression cases fail on the prior implementation and pass with the fix; ownership validation still rejects duplicate claims.
  • python -m pytest -q responses_api_agents/terminus_2_sandboxed_agent/tests --disable-warnings: 135 passed, 4 skipped with Harbor 0.22.0. Align the existing saved-projection test's fallback expectation with helper exclusion while preserving its assertions for producer decisions and command-batch counts (original contract). The earlier CI failure expected six rebuilt turns; only three are retained because three referenced compaction helpers are excluded.
  • All-file pre-commit passed.
  • Real OpenCode 1.17.11 with a controlled local provider: compaction changed from 4 calls / 4 turns to 4 calls / 3 turns; the helper's 511 tokens remain included in the 27,738-token total. The no-compaction control retained 3 calls / 3 turns.

Prepared with Codex. Full repository CI is not yet validated.

Rebase validation — 2026-11-02

Rebased onto main 0c7ac5119c99e0aa0dfeda7be59157c589680ffd, preserving the existing stack order. The assembled original stack passed 1,719 shared conformance, collector, Codex and Kilo tests (16 skipped, 76 configuration-test cases deselected). The separate Terminus saved-projection suite passed all 30 tests. Fresh compaction rollouts retain 16 calls and 15 policy turns on OpenCode, Kilo, Codex, Stirrup and Hermes, including helper usage. All-file pre-commit passed on both stack tips; commits have DCO sign-off and SSH signatures. These are local pinned-runtime runs with a controlled provider, not remote deployment or full repository CI qualification.

@copy-pr-bot

copy-pr-bot Bot commented Oct 27, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working labels Oct 27, 2026
@giuliolovisotto
giuliolovisotto force-pushed the codex/fea-518-collector branch from 8c8c7b8 to 617a5c6 Compare October 29, 2026 11:29
@giuliolovisotto
giuliolovisotto marked this pull request as ready for review October 29, 2026 11:30
@giuliolovisotto giuliolovisotto added the needs-review PR is ready for code review and waiting on a reviewer label Oct 29, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team October 29, 2026 11:31

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

LGTM — no concerns found.

Reviewed the collector changes and regression tests, tracing reference resolution, ownership validation, token accounting, and health consumers. The exclusion preserves captured calls and ownership claims without changing producer-provided trajectories or existing HTTP-error handling.

Static review only; tests were not executed.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

marta-sd
marta-sd previously approved these changes Nov 2, 2026
@giuliolovisotto
giuliolovisotto removed this pull request from stack #5312 November 2, 2026 10:16
@giuliolovisotto
giuliolovisotto removed this pull request from stack #5437 November 2, 2026 10:59
marta-sd
marta-sd previously approved these changes Nov 3, 2026
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
@giuliolovisotto
giuliolovisotto added this pull request to the merge queue Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41797691599

Merged via the queue into main with commit 99ad993 Nov 4, 2026
63 of 66 checks passed
@giuliolovisotto
giuliolovisotto deleted the codex/fea-518-collector branch November 4, 2026 08:31

This branch was previously deployed

1 inactive deployment
public — a3c43dd9 Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #9904
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants