Repository navigation
Conversation
tavily_search keeps one piece of per-session state: the Tavily call metrics that /verify returns. It used the base class's default typed seed, so nothing was fenced or released: the cookie-keyed defaultdict grew for every session, and a late tool call recreated an entry. - seed_session handles the typed body: the session is keyed by resources_session_id, a repeated seed returns it, a seed bound to another episode or a closed session gets 409, and task_data without question or ground_truth gets 422. - close_resources_session releases the metrics by resources_session_id (no cookie needed), is idempotent, and fences late seeds with ResourcesSessionFence. A tool call for a session closed within the window gets 404 instead of recreating its record. - verify keeps a typed session's metrics for its close and releases any other session's, so the /run and MCP paths stop growing the map. - With dump_session_id_to_metrics_on_exit on, released metrics stay until shutdown so the dump still lists every session. The tools and the score read no session state, so an unseeded caller keeps working and a verify for a session the server no longer holds still scores the response (with empty metrics) rather than masking it. Refs GYMINT-227. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
yaoyu-33
added
feature
Contributor
Author
|
/ok to test 421c084 |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-5487-c68eee38630a.docs.buildwithfern.com/nemo/gym |
Every server under the resources session contract answers a verify for a closed session with lost_session_verification, including those whose reward reads only the response, so a closed session is handled the same way everywhere. single_agent_turn verifies before it closes, so the mask only catches an out-of-order verify. TavilySearchVerifyResponse now inherits BaseVerifyResponse so it can carry mask_sample and failure_kind. A session nobody seeded, such as an MCP client's or an Agent's /run, holds no state and is scored as before. Refs GYMINT-227. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tibility path Describe the typed seed as the path new configs use, and name the configs whose agents still call the tools from their own /run, which the server keeps serving until they switch to an Environment Server. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/ok to test 0d1b50b |
yaoyu-33
marked this pull request as ready for review
November 3, 2026 18:33
yaoyu-33
added
the
needs-review
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…he session docs Adds a test that closes a session while web_search awaits Tavily and checks the late search does not recreate its metrics, the property the PR relies on and that a refactor re-indexing the defaultdict after the await would break. Adds a test for a seed that names another task for the same episode (409), and renames the 422 test, which ends with a corrected seed succeeding. Docs: langchain_deepagents_agent calls Tavily with its own client and uses only /seed_session and /verify, so it never reaches the tools and its metrics are always empty. The 409 seed, 404 tool call and masked verify after a close hold for session_close_retry_window_seconds, not forever. The conformance import moves to nemo_gym.server_runtime.testing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 02d74cc |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
zyzhou5
reviewed
Nov 4, 2026
The /run seed guard refused any body sharing a key with ResourcesSeedSessionRequest, so a dataset row with a scalar task_id (HLE rows, gdpval, arc_agi, ...) got 422 on every seed. Treat a body as a malformed typed seed only when it names resources_session_id or carries typed episode_id and task_id objects, the rule #5480 settled on. Also import check_resources_session_contract from nemo_gym.server_runtime, which main's public-API test requires, and correct the docs: when Tavily is only a tool server (HLE opencode_search / pi_search), its /verify and /close_session never reach those sessions, so their metrics stay until shutdown, as on main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test d102e44 |
This was referenced Nov 5, 2026
7 tasks done
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…antics Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
yaoyu-33
changed the base branch from
main
to
yuya/shared-resources-lifecycle
November 5, 2026 23:30
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
…he exit-dump archive Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test 51703ca |
The docs now describe only the typed Environment Server path. The cookie seed still works unchanged; the PR description records that compatibility. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Contributor
Author
|
/ok to test c68eee3 |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Serve typed search sessions and consolidate identity, verification cleanup, close and retirement in #5709. Preserve request-based judge behavior, deep metrics snapshots and optional shutdown metrics dumps.
Depends on #5709 (which depends on #5708). The child diff contains only this environment and its related documentation/configuration; shared implementation is reviewed once in the dependency. No separate issue is needed for this focused migration of an existing resources server.
Retains a deep copy of the metrics in a
finally, also when the judge raised; a retry re-judges from the request.Validation on
51703caaf66e09eab2a3e9947039db23d209f7a6, Python 3.13.14, with this checkout first on PYTHONPATH and NEMO_GYM_EXTRA_ROOTS pointing to it:python -m pytest -o addopts='' -o log_cli=false --import-mode=importlib --tb=short -q resources_servers/tavily_search/tests: 140 passed.ruff check,ruff format --checkandgit diff --checkpassed on the changed files. The private guardwords catalog was unavailable, so that scan was skipped. New commits include DCO sign-off.Real-model smoke on preceding migration revision
739ce5764e034461a4d9cebb57915321a64381c7(shared core4e92d4891) usednvidia/nemotron-3.5-lightning-30b-a3b, Python 3.13.14, live Tavily searches and the actual model judge. Four searches succeeded and six model calls completed; the judge rejected the model’s incorrect answer, returning reward 0.0 unmasked, and live metrics/identity state was released. The observed score is retained rather than rerun for a better answer. The diagnostic uses a task-only empty exclusion policy and an 8192-token judge cap; it validates provider/judge/lifecycle plumbing, not benchmark accuracy or exclusion-policy effectiveness. Loaded source hashes matched that revision. Deep snapshot retries, cleanup failures and exit dumping are covered by regressions. The cookie MCP path used by the HLE configs is covered by unit tests only. For a rejected answer the smoke row'sextracted_final_answerisno; an empty string would mean the judge text did not parse, which_grade_samplescores as 0 unmasked (pre-existing).These targeted checks do not replace the repository CI matrix or approval of the current head.