Repository navigation
test(conformance): enable shared compaction scenario - #5439
Conversation
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
SHIP
No blocking issues found. One documentation NOTE below would align the published capability matrix with the newly enabled scenario.
I reviewed all three changed files and the affected configuration, evidence-checking, gating, and reporting paths. This was a static review; I couldn't execute tests or independently reproduce the reported live runs.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Inline comments (escaped v1):
-
fern/versions/latest/pages/reference/trajectory-capabilities.mdxline 13: NOTE — Suggestion: update the published compaction scope. The new catalog entry makes the CI renderer include compaction measurements, but this page still excludes compaction from qualification. Readers will see the new results beneath that contradictory scope statement. The publisher preserves surrounding prose, so regeneration won't resolve it.Could you replace lines 12–13 with the following, retaining the other exclusions and the README's measurement limits?
The checks cover local execution, including P1 context-compaction evidence checks. Remote sandboxes, agent sessions, multimodal inputs, and parallel execution require separate qualification. Compaction usage is synthetic; these checks do not measure compression quality or before/after context token counts, or validate boundary references.
Acceptance — proposed static check: after table insertion, the page should describe compaction evidence as included while retaining the other exclusions and explicitly avoiding compression-quality, token-reduction, or boundary-reference qualification claims. This is a documentation suggestion, not an executed validation result.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
40562f8 to
0750d90
Compare
|
🌿 Preview your docs: https://nvidia-preview-pr-5439-3f0acfff49e2.docs.buildwithfern.com/nemo/gym |
0750d90 to
bddc418
Compare
12de7ed to
0578906
Compare
a3e3b5b to
8d44e03
Compare
8d44e03 to
8fa40d4
Compare
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/41807705192 |
Register one shared
context_compactionscenario after the collector, shared checker/provider support and Kilo/Codex/Stirrup/Hermes fixes. This is the final PR in the original seven-PR stack, based on #5311. The separate Pi follow-up, #5435 → #5436, stays above this PR. This completes the scenario introduced by #5307; no separate issue is needed.The task runs fourteen real shell actions with small context/threshold settings. Every selected harness faces the same five checks: completed event, summary reuse in the next policy request, helper references and ownership, exclusion from policy turns, and retained usage. SimpleAgent remains unchanged. Earlier layers retain the eight-scenario catalog. The documentation now includes P1 compaction evidence while preserving the measurement limits.
Validation after rebasing onto main
0c7ac5119c99e0aa0dfeda7be59157c589680ffd:--gate-tier P1. Fresh OpenCode, Kilo, Stirrup and Hermes compaction scenarios pass too; each retains 16 calls, 15 policy turns and one event/helper.npm --prefix fern run check: 0 errors, 1 warning. Commits have DCO sign-off and SSH signatures.Reproduce with the selected harness dependencies installed:
Default CI reports all nine scenarios and retains the eight-scenario P0 gate. Explicit P1 gating also requires compaction. Usage is synthetic fixture input; compression quality, before/after counts and boundary references are outside these five checks. Runs use real pinned runtimes and a controlled local provider; remote deployments and full repository CI were not qualified.
Continues #5123, whose prior discussion remains available there.