Skip to content

test(e2e): add Deep Agents Code headless inference acceptance check (#5619) - #5652

Closed
abhi-0906 wants to merge 1 commit into
NVIDIA:mainfrom
abhi-0906:feat/issue-5619-headless-check
Closed

abhi-0906 wants to merge 1 commit into
NVIDIA:mainfrom
abhi-0906:feat/issue-5619-headless-check

Conversation

@abhi-0906

@abhi-0906 abhi-0906 commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5619

Adds test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh, a skip-aware live check (mirroring the existing 05/06 Deep Agents Code checks) for headless dcode -n:

  • config.toml routes through the managed https://inference.local endpoint.
  • Headless dcode -n "<deterministic prompt>" returns a response or a deterministic, actionable provider/model error within a timeout — never a hang.
  • No provider or proxy credentials appear in config.toml, .env, .mcp.json, /tmp/nemoclaw-proxy-env.sh, or the captured output.

The check self-skips when the sandbox is not a Deep Agents Code sandbox. A unit assertion in langchain-deepagents-code-image.test.ts registers the check and verifies its skip guard, prompt, inference route, and secret scan.

Signed-off-by: Abhimanyu Kumar abhimanyukumar7290@gmail.com

…VIDIA#5619)

Add a skip-aware live check (07-deepagents-code-headless-inference.sh) that runs
`dcode -n` inside a built Deep Agents Code sandbox and asserts:

- config.toml routes through the managed https://inference.local endpoint
- headless `dcode -n` returns a deterministic response or actionable provider/model
  error within a timeout (no hang/ambiguous failure)
- no real provider/proxy credentials (nvapi-/sk-/xox.-/AKIA shapes) appear in
  config.toml, .env, .mcp.json, /tmp/nemoclaw-proxy-env.sh, or the captured output

The script self-skips when the sandbox is not a Deep Agents Code sandbox, mirroring
the existing 05/06 checks. A unit assertion in the image test registers the check
and verifies its skip guard, prompt, inference route, and secret-scan content.

The live green run requires a built sandbox plus the managed inference endpoint and
is gated to the live e2e environment.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Jun 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

This repository limits contributors to 10 open pull requests. Please close or merge existing PRs before opening new ones.

@github-actions github-actions Bot closed this Jun 23, 2026
@coderabbitai

coderabbitai Bot commented Jun 23, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: af0ecf13-9b87-4ff1-a5fe-c44064f44a64

📥 Commits

Reviewing files that changed from the base of the PR and between a9f31e4 and 28e3a12.

📒 Files selected for processing (2)
  • test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh
  • test/langchain-deepagents-code-image.test.ts

📝 Walkthrough

Walkthrough

Adds a new Bash e2e check script (07-deepagents-code-headless-inference.sh) that skips when outside a Deep Agents Code sandbox, validates config.toml routes through inference.local, runs dcode -n and checks for actionable output, scans artifacts for secret patterns, and exits non-zero on failure. A companion Vitest test asserts the script contains required markers.

Changes

Deep Agents Code headless inference e2e check

Layer / File(s) Summary
Headless inference check script
test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh
New Bash script with strict options, skip guard for non-DA-Code sandboxes, config.toml routing assertion against inference.local, dcode -n execution with timeout and exit-code handling, secret-pattern leak scan over config/runtime/output artifacts, and final PASS/FAIL summary with non-zero exit on any failure.
Vitest content assertion
test/langchain-deepagents-code-image.test.ts
Adds a new it(...) test that reads the check script and asserts presence of dcode headless invocation, inference.local, nvapi- marker, and /tmp/nemoclaw-proxy-env.sh.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

Poem

🐇 A sandbox appears, so I sniff all around,
Does dcode -n run? Does inference.local sound?
No nvapi- keys left out in the air,
No secrets in .env or config to share.
PASS or FAIL — the rabbit counts each,
And exits with one if the tests breach! 🌿

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@wscurran wscurran added the integration: dcode LangChain Deep Code integration behavior label Jun 23, 2026
cv added a commit that referenced this pull request Jun 26, 2026
…5619) (#5789)

Closes #5619
Supersedes #5652

Adds
`test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh`,
a skip-aware live check (mirroring the existing `05`/`06` Deep Agents
Code checks) for headless `dcode -n`:

- `config.toml` routes through the managed `https://inference.local`
endpoint.
- Headless `dcode -n "<deterministic prompt>"` returns a response or a
deterministic, actionable provider/model error within a timeout — never
a hang.
- No provider or proxy credentials appear in `config.toml`, `.env`,
`.mcp.json`, `/tmp/nemoclaw-proxy-env.sh`, or the captured output.

The check self-skips when the sandbox is not a Deep Agents Code sandbox.
A unit assertion in `langchain-deepagents-code-image.test.ts` registers
the check and verifies its skip guard, prompt, inference route, and
secret scan.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added an end-to-end Bash check for “Deep Agents Code” headless
inference in a managed sandbox.
* Validates routing to the expected local inference endpoint and use of
managed placeholder API key references.
* Improves headless execution verification with explicit timeout
handling and deterministic success response checks.
* Rejects ambiguous output and local-failure style outcomes; adds
classification coverage for pass/actionable errors/timeouts.
* Adds secret-like value scanning across sandbox config and captured
runtime/proxy artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
…VIDIA#5619) (NVIDIA#5789)

Closes NVIDIA#5619
Supersedes NVIDIA#5652

Adds
`test/e2e/e2e-cloud-experimental/checks/07-deepagents-code-headless-inference.sh`,
a skip-aware live check (mirroring the existing `05`/`06` Deep Agents
Code checks) for headless `dcode -n`:

- `config.toml` routes through the managed `https://inference.local`
endpoint.
- Headless `dcode -n "<deterministic prompt>"` returns a response or a
deterministic, actionable provider/model error within a timeout — never
a hang.
- No provider or proxy credentials appear in `config.toml`, `.env`,
`.mcp.json`, `/tmp/nemoclaw-proxy-env.sh`, or the captured output.

The check self-skips when the sandbox is not a Deep Agents Code sandbox.
A unit assertion in `langchain-deepagents-code-image.test.ts` registers
the check and verifies its skip guard, prompt, inference route, and
secret scan.

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Added an end-to-end Bash check for “Deep Agents Code” headless
inference in a managed sandbox.
* Validates routing to the expected local inference endpoint and use of
managed placeholder API key references.
* Improves headless execution verification with explicit timeout
handling and deterministic success response checks.
* Rejects ambiguous output and local-failure style outcomes; adds
classification coverage for pass/actionable errors/timeouts.
* Adds secret-like value scanning across sandbox config and captured
runtime/proxy artifacts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Abhimanyu Kumar <abhimanyukumar7290@gmail.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

integration: dcode LangChain Deep Code integration behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Validate Deep Agents Code headless dcode inference

2 participants