[devin-ai-integration]

Fixes OR-631.

Summary

• Root cause: the SSH helper's log_chunk (src/jobs/ssh/helper.py) runs on the login node. Its first probe of <run>/log, made before the compute node creates it, leaves an NFS negative-lookup cache entry. The cache can last for the directory's attribute-cache window (up to 60 s by default). A job that fails quickly creates and writes log inside that window, so every collect still gets {"missing": true}. Then run_slurm's 20 s terminal drain in supervise.rs times out and the run ends with an empty local log, even though the remote log has content.
• Fix: when open raises FileNotFoundError, list the run directory once and retry the open. On Linux NFS, opening a directory revalidates its attributes, which drops the stale negative entries. If the log really is absent, the helper still returns missing.

Test plan

• Reproduced on unmodified main (Linux, local Slurm 21.08 with /scratch NFS-mounted from the same host, OpenResearch 0.2.21). I ran orx exp run … --backend slurm on a train.py that fails an assert right away. The run ended Failed and the remote log held 330 bytes with the traceback. orx logs said no log was captured and the dashboard log pane was empty. The supervisor log showed Remote log is not available yet until the drain gave up.
• On a fresh dev slot built from this branch, a live Claude Code dashboard chat replied. I then ran the same Slurm experiment. orx logs and the dashboard log pane both show the full 330-byte traceback.
• Before the fix, I checked the same thing directly on this NFS mount: after a cached miss, a plain re-open still failed, while os.listdir(dir) followed by open succeeded.
• scripts/test_ssh_helper.py gains test_stale_missing_log_is_revalidated_by_listing_the_run_directory: the first open hits a cached miss, listdir gets the run directory, and the retry returns the log bytes. It fails on main and passes here.
• cargo build --locked, cargo fmt --check and cargo clippy --locked -- -D warnings pass. In cargo test --locked, 1276 tests passed and 3 failed in local::projects::tests. They fail because this VM has a global git identity that differs from the one those tests expect; the failures are unrelated to this change.

Automation improvements

• CODEX_AUTH_JSON is stale (refresh token already used, 401), so Codex live preflight could not run. Refresh the secret from a new ChatGPT Codex login so both agents can be verified.
• Add Slurm + munge + an NFS loopback export (/srv/export → /scratch, with HWLOC_SYNTHETIC for slurmd) to the VM blueprint. That would turn about an hour of Slurm-bug setup into minutes.

Link to Devin session: https://app.devin.ai/sessions/37aa99ba59484e86bec3eb67a115cd4e
Open in Devin Desktop: https://app.devin.ai/desktop/session/37aa99ba59484e86bec3eb67a115cd4e?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

The helper lists the run directory and retries before reporting a missing log.

• SSH log collection retries after a missing-file lookup.

_{Reviews (2) · Last reviewed commit: "Assert the retry lists the run directory..." · Reviewed by Greptile}

[greptile-apps]

Successful retry has no test

The new retry-success path has no automated test. The existing missing-log test deletes the file, so both opens fail; it would still pass if the retry were removed. That leaves this fix unprotected against a later change.

Add a test that makes the first open raise FileNotFoundError, checks that os.listdir receives the run directory, and lets the second open read a real log. Assert that the returned bytes match the log.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/jobs/ssh/helper.py
Line: 99-103

Comment:
**Successful retry has no test**

The new retry-success path has no automated test. The existing missing-log test deletes the file, so both opens fail; it would still pass if the retry were removed. That leaves this fix unprotected against a later change.

Add a test that makes the first `open` raise `FileNotFoundError`, checks that `os.listdir` receives the run directory, and lets the second `open` read a real log. Assert that the returned bytes match the log.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[devin-ai-integration]

Added in 094cc0c and tightened in 2b6a7e2: test_stale_missing_log_is_revalidated_by_listing_the_run_directory makes the first open raise FileNotFoundError, asserts os.listdir gets the run directory, and checks the retried read returns the log bytes. It fails on main and passes with the fix.

[devin-ai-integration]

@greptileai review

[paperclover]

superseded by #862.

[devin-ai-integration]

Noted. One caveat: #862's description lists "OR-631: NFS log lookup unchanged". If orx-host still reads <run>/log from an NFS-mounted login node after an early miss, the stale negative-lookup cache can still hide a fast-failing job's log. If the cause isn't removed there, I can port this one-line revalidation and its test (test_stale_missing_log_is_revalidated_by_listing_the_run_directory) onto #862 or a follow-up branch.