fix(e2e): add docker build retry for transient registry timeouts

[hunglp6d]

Fixes #4696

✨ [AI-generated PR]

Summary

The rebuild-hermes-e2e / run job failed in
nightly-e2e run #26856257187
(job #79199640102)
with a Docker Hub registry timeout:

dial tcp 54.224.69.1:443: i/o timeout
ERROR: failed to build: failed to solve: DeadlineExceeded: node:22-trixie-slim@sha256:2d9f…

Root cause

Docker Hub (registry-1.docker.io) was transiently unreachable from the GitHub-hosted
runner during Phase 2 ("Building Hermes base image with v2026.4.13"). The docker build
command had no retry logic, so a single transient network error failed the entire test.

Fix

Wrap both docker build invocations in test/e2e/test-rebuild-hermes.sh with a
3-attempt retry loop and exponential backoff (10 s, 20 s, 30 s):

• Phase 2 — old Hermes base image build (nemoclaw-hermes-old-base:e2e-rebuild)
• Phase 5 — current Hermes base image build (ghcr.io/nvidia/nemoclaw/hermes-sandbox-base:latest)

Both hit Docker Hub for the node:22-trixie-slim digest-pinned base image and are
equally vulnerable to transient registry outages.

Validation

The GitHub token used for this PR does not have the workflow scope required to push a
custom-e2e.yaml validation branch. To validate the fix, re-run the
rebuild-hermes-e2e job from this branch:

gh workflow run nightly-e2e.yaml --repo NVIDIA/NemoClaw \
  --ref fix/nightly-e2e-docker-registry-timeout-retry-f90000b \
  -f jobs=rebuild-hermes-e2e

• Original failing run: #26856257187 on f90000b

Test plan

• Re-run rebuild-hermes-e2e on this branch and confirm the retry loop handles a transient registry timeout
• Verify no regressions in the rebuild flow (marker file survival, version upgrade, credential preservation)

Signed-off-by: Hung Le hple@nvidia.com

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 187379df-957b-4d11-b6bd-d240d500ea7a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/nightly-e2e-docker-registry-timeout-retry-f90000b

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

E2E Advisor Recommendation

Required E2E: None
Optional E2E: rebuild-hermes-e2e, rebuild-hermes-stale-base-e2e

Dispatch hint: rebuild-hermes-e2e,rebuild-hermes-stale-base-e2e

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

• None. No merge-blocking E2E is required because this PR changes only an existing E2E test script and cannot affect runtime/user flows directly.

Optional E2E

• rebuild-hermes-e2e (high): Optional validation that the modified test script still exercises the standard Hermes rebuild upgrade path; this path uses both the old-base and current-base Docker build retry blocks touched by the PR.
• rebuild-hermes-stale-base-e2e (high): Optional validation that the modified test script still exercises the stale cached Hermes base-image rebuild path; this path uses the changed old-base Docker build retry logic.

New E2E recommendations

• None.

Dispatch hint

• Workflow: .github/workflows/nightly-e2e.yaml
• jobs input: rebuild-hermes-e2e,rebuild-hermes-stale-base-e2e

[github-actions]

E2E Scenario Advisor Recommendation

Required scenario E2E: None
Optional scenario E2E: None

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

• None. Changed file is under legacy test/e2e/ and outside the scenario E2E suite/workflows; it does not directly affect test/e2e-scenario/ behavior.

Optional scenario E2E

• None.

Relevant changed files

• None.

[github-actions]

PR Review Advisor

Findings: 0 needs attention, 2 worth checking, 1 nice ideas
Top item: Retry workaround lacks direct coverage/removal criteria

Review findings

🛠️ Needs attention

• None.

🔎 Worth checking

• Source-of-truth review needed: Docker build retry loops in Hermes rebuild E2E: The advisor marked localized patch analysis as needs_followup.
  • Recommendation: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Evidence: Lines 160-175 and 303-315 add inline retry loops around the two Docker builds, with comments identifying transient Docker Hub registry timeouts as the reason.
• Retry workaround lacks direct coverage/removal criteria (test/e2e/test-rebuild-hermes.sh:160): The PR adds recovery behavior for transient external Docker registry failures, but the retry branch is only exercised if the live E2E happens to hit a real transient failure. There is also no in-code note for when this workaround can be removed, such as after reliable registry mirroring/caching is introduced.
  • Recommendation: Consider documenting the external source of the invalid state and the removal condition near the retry logic. If this retry pattern is expected to grow, factor it into a small helper that can be tested with a failing-then-succeeding command stub.
  • Evidence: The old-base build and current-base build now use inline `for _attempt in 1 2 3` loops with sleeps at lines 160-175 and 303-315; no deterministic injected-failure test or helper-level coverage accompanies the recovery behavior.

🌱 Nice ideas

• Final failed build attempt still logs and sleeps as if another retry remains (test/e2e/test-rebuild-hermes.sh:172): On the third failed attempt, the script prints `retrying in 30s...` and sleeps for 30 seconds before immediately failing, which adds avoidable delay and misleading output for deterministic Dockerfile/build failures.
  • Recommendation: Only log and sleep when `_attempt` is less than the maximum attempt count, or change the final-attempt message to indicate that no retries remain.
  • Evidence: Both retry loops unconditionally run `info "docker build attempt ${_attempt}/3 failed; retrying in $((10 * _attempt))s..."` followed by `sleep` after any failed attempt, including attempt 3.

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

[wscurran]

✨
Related open issues:

• #4696 fix(e2e): Docker Hub registry timeout fails rebuild-hermes-e2e with no retry

[hunglp6d]

Closed due to fixed