[ffrujeri]

Stacked on #5475 (the Lite and Full recipe), in the BIRD-Interact stack: #5465 → #5409 → #5473 → #5475 → this PR. The diff against #5475 is the bird_interact Environment Server, the recipe's prepare.py and README, and ATTRIBUTIONS.md.

What changed

auto-ontology-eval runs BIRD-Interact's Lite and Full query tasks in c-Interact on the BIRD-Interact ADK: its orchestrator, its user simulator and its grader. This PR brings the bird_interact Environment Server to parity with what it uses, at ADK revision 451fe2c3518ee1cf908d8139e2913483bd519381.

ADK	Gym, after this PR
orchestrator/cinteract.py: max_turn, the phase-1, debug and follow-up messages, two runs for each phase	app.py: _first_request, _next_message
system_agent/agent.py, tools.py, callbacks_cinteract.py: the instruction, ask_user and submit_sql, one submission for each run	app.py: CInteractTools
db_environment/server.py: the grader's messages, rewards by submission count, the phase-1 snapshot, the knowledge the agent sees	app.py: submit; grading.py
shared/db_utils.py: test_case_default and its rewrites	grading.py
user_simulator/: the two stages, the v2 prompts, the clause split (sqlglot)	simulator.py

• c-Interact is the ADK's protocol. The agent gets the two tools, served over direct_http or mcp. The server sends the ADK's instruction and phase-1 message, then opens each later turn as the orchestrator does: the debug message after a failed phase (with the SQL error when the submission did not run), the follow-up question after phase 1 passes. A turn ends at its submission; a later call in the turn gets the ADK agent's reply. Rewards follow the phase's submissions, as in the ADK. This replaces feat(bird_interact): run BIRD-Interact in its Environment Server, with no Resources Server #5409's conversational c-Interact, where the agent's reply text was parsed for SQL. max_turns is gone: the protocol fixes the turns.
• The user simulator is the ADK's. Its v2 prompts, the action parser and the response generator at temperature 0 with its output limits, the ambiguity JSON as the row writes it, the solution split into clauses with sqlglot, and phase 2's prompts. The one deliberate difference: the ADK fills its placeholders one after another, so an agent's question containing [[GT_SQL]] would get the solution SQL written into the simulator's prompt; Gym fills them in one pass.
• Grading is the ADK's default test case. The submission as written (its executability), then the submission and the solution with the ADK's comment, DISTINCT and ROUND rewrites, in that order on one database state, rolled back; each query keeps its last statement's rows, at most 10,000. Edge cases are kept for parity, such as an array of decimals failing because it cannot be written as JSON. A PostgreSQL query is sent whole, so dollar-quoted bodies work. Phase 1's passing SQL is applied for phase 2, and replayed when a sandbox is replaced. sql_timeout_seconds defaults to the ADK's 60.
• The category rule. A query phase is graded by its result even when it has test cases; only a phase of another category with test cases (the management tasks) is refused at seed, and prepare.py leaves it out. Every query task's follow-up is a query, so all 195 Lite and 410 Full query tasks run.
• Simulator model. Still user_model (user_model_name in env.yaml). The READMEs say that the ADK default, which auto-ontology-eval uses, is anthropic/claude-haiku-4-5-20251001.

Why

auto-ontology-eval's BIRD-Interact results come from the ADK. With the same protocol, simulator and grading, its runs can move to Gym without a change in what is measured. Tracking: GYMINT-646 and GYM-1103 (Linear).

Code provenance

environment_servers/bird_interact/adk_prompts.py holds texts copied verbatim from the BIRD-Interact ADK, revision 451fe2c3518ee1cf908d8139e2913483bd519381, under the MIT license (Copyright (c) 2025 BIRD-Interact Authors), whose notice the file's header reproduces:

Upstream file	Copied into adk_prompts.py
BIRD-Interact-ADK/user_simulator/prompts.py	v2_action_parser, v2_response_generator
BIRD-Interact-ADK/user_simulator/server.py	the fallback reply
BIRD-Interact-ADK/system_agent/agent.py	CINTERACT_INSTRUCTION
BIRD-Interact-ADK/system_agent/tools.py	the ask_user and submit_sql docstrings, the submission reply's lines
BIRD-Interact-ADK/system_agent/callbacks_cinteract.py	the clarification note, the replies after a submission
BIRD-Interact-ADK/orchestrator/cinteract.py	the phase-1, debug and follow-up messages
BIRD-Interact-ADK/db_environment/server.py	the grader's messages, the visible knowledge fields

The texts are byte-for-byte; the ADK's f-strings are str.format templates. ATTRIBUTIONS.md records the component. The code that uses them (protocol, simulator pipeline, grading) is an independent implementation.

Validation

Check	Result
pytest environment_servers/bird_interact/tests (server venv)	64 passed, 4 skipped: the real-PostgreSQL class, which needs initdb on the host (run in the image instead, below)
pytest benchmarks/bird_interact/tests	7 passed, including the category rule
Grading parity: the ADK's own test_case_default and submit flow against grading.py, in the Lite image	863 of 863 submissions got the same verdict, on six Lite databases (544 passed, 247 not correct, 72 execution errors): DISTINCT and ROUND rewrites, comments, ordering, decimals, dates, JSON, numeric and date arrays, multi-statement and writing submissions
grading.py in the Lite image	setup with replayed SQL, dollar-quoted bodies, apply on pass, timeouts and a broken solution, on real PostgreSQL
Mutation check	Dropping the replay of phase 1's SQL on a fresh sandbox fails test_phase_2_sees_what_phase_1_changed_even_on_a_fresh_sandbox
tests/unit_tests (benchmark, manifest, import, sandbox and Environment Server selections)	21 failures in test_opensandbox_compose.py, test_sandbox_transfer.py, test_sandbox_stop_retry.py, test_harbor_validation.py and test_benchmarks.py, the same with and without this PR's changes on this machine
pre-commit run --files on the changed files	Passed

Not validated yet: runs with the real ground truth (GYMINT-592); the Full image build and a Full run; a run with the ADK's simulator model.

Rollout evidence

All on Docker sandboxes, with simple_agent; policy azure/openai/eccn-gpt-6.1-sol, simulator openai/openai/eccn-gpt-6-luna:

• The server's five example tasks (SQLite), both modes: 10 of 10 completed, mean reward 0.82; they replace data/example_rollouts.jsonl. c-Interact took each path but phase 2's debug turn: questions with the clarification note, a failed submission and its debug turn passing for 0.5, the follow-up as phase 2.
• Lite through the recipe (PostgreSQL, the image from feat(benchmarks): add the BIRD-Interact Lite and Full benchmark recipe #5475), three tasks with a stand-in ground truth written from their clear questions and the real schema, because the real one is not in hand. This checks the image, the protocol and the grading on Lite; its rewards are not results. c-Interact: solar_10 passed both phases (1.0); solar_3 passed phase 1, failed phase 2 and its debug turn (0.7); crypto_2 failed phase 1 and its debug turn (0.0). A second submit_sql in one turn got the ADK's "SQL submitted. Awaiting result." a-Interact: 3 of 3 completed, mean 0.9.

One simulator reply in the example run was the action itself (unlabeled("JOIN")): the simulator model continued the ADK's prompt, which ends in an open <s>, with the action. The ADK reads it the same way.

Compatibility

Within the unmerged stack: c-Interact's protocol, stop reasons and grading change, so c-Interact scores from #5409's server are not comparable with these. max_turns is removed from the server's config. c-Interact now needs a harness that resumes its session and takes tool grants: simple_agent does; OpenCode, which ran #5409's c-Interact, refuses required tool grants. a-Interact keeps its tools and budget; its submit now replies with the grader's messages. New dependency for the server: sqlglot==30.20.0.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ffrujeri]

/ok to test cb77043

[ffrujeri]

/ok to test 478d3bc

[ffrujeri]

/ok to test 33211b2