Skip to content

test(conformance): enable shared compaction scenario - #5439

Merged
giuliolovisotto merged 3 commits into
codex/fea-518-hermesfrom
codex/fea-518-compaction
Nov 4, 2026
Merged

giuliolovisotto merged 3 commits into
codex/fea-518-hermesfrom
codex/fea-518-compaction

Conversation

@giuliolovisotto

@giuliolovisotto giuliolovisotto commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Register one shared context_compaction scenario after the collector, shared checker/provider support and Kilo/Codex/Stirrup/Hermes fixes. This is the final PR in the original seven-PR stack, based on #5311. The separate Pi follow-up, #5435 → #5436, stays above this PR. This completes the scenario introduced by #5307; no separate issue is needed.

The task runs fourteen real shell actions with small context/threshold settings. Every selected harness faces the same five checks: completed event, summary reuse in the next policy request, helper references and ownership, exclusion from policy turns, and retained usage. SimpleAgent remains unchanged. Earlier layers retain the eight-scenario catalog. The documentation now includes P1 compaction evidence while preserving the measurement limits.

Validation after rebasing onto main 0c7ac5119c99e0aa0dfeda7be59157c589680ffd:

  • Shared conformance, collector, Codex and Kilo tests: 1,719 passed, 16 skipped, 76 configuration-test cases deselected. Terminus saved-projection tests: 30 passed.
  • Fresh Codex: all nine scenarios pass with --gate-tier P1. Fresh OpenCode, Kilo, Stirrup and Hermes compaction scenarios pass too; each retains 16 calls, 15 policy turns and one event/helper.
  • Before this rebase, a full Pi run on this layer passed the eight P0 scenarios and reported the missing compaction event. The separate Pi follow-up passes all nine scenarios after this rebase. SimpleAgent still has no compaction support.
  • All-file pre-commit passes on both stack tips. npm --prefix fern run check: 0 errors, 1 warning. Commits have DCO sign-off and SSH signatures.

Reproduce with the selected harness dependencies installed:

python scripts/run_harness_conformance.py --harness codex --scenario context_compaction --gate-tier P1 --output results/codex-compaction

Default CI reports all nine scenarios and retains the eight-scenario P0 gate. Explicit P1 gating also requires compaction. Usage is synthetic fixture input; compression quality, before/after counts and boundary references are outside these five checks. Runs use real pinned runtimes and a controlled local provider; remote deployments and full repository CI were not qualified.

Continues #5123, whose prior discussion remains available there.

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SHIP

No blocking issues found. One documentation NOTE below would align the published capability matrix with the newly enabled scenario.

I reviewed all three changed files and the affected configuration, evidence-checking, gating, and reporting paths. This was a static review; I couldn't execute tests or independently reproduce the reported live runs.

Note: formal approval is not available for this request, so this review is posted as a comment only.


Inline comments (escaped v1):

  • fern/versions/latest/pages/reference/trajectory-capabilities.mdx line 13: NOTE — Suggestion: update the published compaction scope. The new catalog entry makes the CI renderer include compaction measurements, but this page still excludes compaction from qualification. Readers will see the new results beneath that contradictory scope statement. The publisher preserves surrounding prose, so regeneration won't resolve it.

    Could you replace lines 12–13 with the following, retaining the other exclusions and the README's measurement limits?

    The checks cover local execution, including P1 context-compaction evidence checks.
    Remote sandboxes, agent sessions, multimodal inputs, and parallel execution require
    separate qualification. Compaction usage is synthetic; these checks do not measure
    compression quality or before/after context token counts, or validate boundary references.

    Acceptance — proposed static check: after table insertion, the page should describe compaction evidence as included while retaining the other exclusions and explicitly avoiding compression-quality, token-reduction, or boundary-reference qualification claims. This is a documentation suggestion, not an executed validation result.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

@giuliolovisotto giuliolovisotto added area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters ci CI, automation, test queue, or workflow infrastructure work needs-review PR is ready for code review and waiting on a reviewer labels Nov 2, 2026
@giuliolovisotto
giuliolovisotto force-pushed the codex/fea-518-compaction branch from 40562f8 to 0750d90 Compare November 2, 2026 12:18
@nemo-automation-bot
nemo-automation-bot Bot requested a review from a team November 2, 2026 12:19
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

@giuliolovisotto
giuliolovisotto force-pushed the codex/fea-518-compaction branch from 12de7ed to 0578906 Compare November 3, 2026 10:13
@giuliolovisotto
giuliolovisotto force-pushed the codex/fea-518-compaction branch from a3e3b5b to 8d44e03 Compare November 3, 2026 14:15
@giuliolovisotto
giuliolovisotto enabled stacked auto-merge November 3, 2026 14:27
@giuliolovisotto
giuliolovisotto force-pushed the codex/fea-518-compaction branch from 8d44e03 to 8fa40d4 Compare November 3, 2026 15:11
marta-sd
marta-sd previously approved these changes Nov 4, 2026
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41807705192

This branch was previously deployed

1 inactive deployment
public — 3f0acfff Deployed Nov 4, 2026 by copy-pr-bot[bot] via release / finalize / notify #9928
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:evaluation Evaluation, rollout collection, reward profiling, comparison, and exporters ci CI, automation, test queue, or workflow infrastructure work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants