Skip to content

fix(enterpriseops): preserve rollout sessions and clean up failed seeds - #4942

Closed
cmunley1 wants to merge 3 commits into
mcuevas-nvidia-bench-integration-enterpriseops-signedfrom
codex/fix-enterpriseops-review-blockers
Closed

cmunley1 wants to merge 3 commits into
mcuevas-nvidia-bench-integration-enterpriseops-signedfrom
codex/fix-enterpriseops-review-blockers

Conversation

@cmunley1

@cmunley1 cmunley1 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The optional EnterpriseOps telemetry agent loses its resource-session cookie when a tool returns no Set-Cookie, and keeps generating after a completed response contains only reasoning or no output. Partial database-seeding failures also leave successful sibling databases without a cleanup owner. This follow-up fixes those paths and the parent PR's failing config/test integration.

Targets the head branch of #2142 so the fixes can be incorporated before that benchmark merges. A separate issue is unnecessary: these are reproduced findings from that PR's review.

Changes

  • Preserve resource cookies across tool calls, including cookie rotation, through /verify. Stop reasoning-only/empty responses using the existing SimpleAgent termination semantics while retaining turn telemetry and usage.
  • Include Daniele Molinari's cleanup fix from fix(enterpriseops): retain databases until cleanup succeeds #4774 (7520e387a), preserving its author and sign-off: wait for all seed attempts, retain failed deletions for retry, and prevent reseeding from discarding database handles.
  • Declare Ray opt-out for both new server classes and explicit environment-server relays for both agent configurations. Give the telemetry overlay a distinct relay name so composing it does not orphan the base agent.
  • Replace private HTTP-client cookie-jar manipulation in the session-isolation tests with public cookie/header APIs, compatible with both the locked dependencies and CI's httpx2==2.13.1 path.

The default SimpleAgent and SQL scoring formulas are unchanged. The optional telemetry agent now stops invalid terminal generations instead of making extra model calls. Reseeding after a failed cleanup reports an error rather than losing the outstanding database handles.

Validation

All checks ran against the final diff on Python 3.13.14:

  • gym env test +entrypoint=resources_servers/enterpriseops_gym +should_validate_data=true +skip_venv_if_present=true: 115 passed, 1 skipped, example-data validation passed. Ran with locked dependencies and again with CI's httpx2==2.13.1; both passed. The skip is the optional tool-snapshot checksum check because snapshots were not prepared.
  • gym env test +entrypoint=responses_api_agents/turn_logging_agent +should_validate_data=true +skip_venv_if_present=true: 7 passed.
  • python -m pytest -q -o addopts= --import-mode=importlib tests/unit_tests/test_ray_opt_out.py tests/unit_tests/test_add_legacy_agent_environment_servers.py tests/unit_tests/test_apptainer_provider.py: 140 passed, including all three previously failing root integration tests.
  • All four new agent regression cases fail against the original parent implementation. The strict config-composition regression also reproduced the relay-name collision before its fix.
  • Independent partial-seed probes cover either completion ordering: one seeded database is deleted, with zero orphaned databases.
  • pre-commit run --all-files and git diff --check: passed. All commits include DCO sign-off.

Rollout evidence

An integration smoke against the exact pinned CSM Docker image advertised 89 tools, seeded the native example database, and successfully queried its 18 tables. Injecting a first deletion failure retained the database handle; the subsequent janitor pass performed the real deletion and removed the session. This validates the service and cleanup path, but is not a model rollout.

Draft pending a representative real-model smoke, as required by AGENTS.md. The saved model configuration returns HTTP 401 on both model discovery and a minimal inference request; no model rollout completed on this diff. The older full-benchmark run is not counted as validation of these changes. Managed Apptainer startup has not been revalidated by this follow-up.

d-molinari and others added 3 commits October 22, 2026 13:13
Signed-off-by: Daniele Molinari <dmolinari@nvidia.com>
(cherry picked from commit 7520e38)
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
Signed-off-by: Christian Munley <cmunley@nvidia.com>
@cmunley1 cmunley1 added bug Something isn't working area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers labels Oct 22, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@cmunley1 cmunley1 closed this Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants