Remove nvsnap-cuda-checkpoint CLI now that upstream ships arm64

[balajinvda]

NVIDIA/cuda-checkpoint now publishes an aarch64 prebuilt (bin/aarch64_Linux), which removes the reason we built our own CLI.

Background

docker/agent/nvsnap-cuda-checkpoint.c is a drop-in replacement for the upstream cuda-checkpoint binary, built on the public driver checkpoint API (cuCheckpointProcess*). Dockerfile.base records two reasons for it:

buildable for x86-64 AND arm64 (NVIDIA publishes x86-64 only), and extensible (r580 gpuPairs migration, lock timeouts)

Reason 1 is now resolved upstream. Reason 2 is a judgement call (see below).

Cleanup scope

• docker/agent/nvsnap-cuda-checkpoint.c
• cuda-cli-builder stage in docker/agent/Dockerfile.base
• copy logic in scripts/build-agent.sh (~lines 148-170)
• docker/agent/cuda-checkpoint-wrapper.sh (re-evaluate; it points LD_LIBRARY_PATH at the node driver libcuda)

Blockers to check before removing

1. Driver floor on arm. Upstream lists ARM CPU support as r595+. Our Dockerfile targets r570+ and the CLI header notes 550+ for basic actions. Switching to the upstream arm binary raises the arm driver requirement to r595. Inventory arm node driver versions first; if any sit below r595 this trades a working path for a broken one.

2. glibc compatibility. Commit 4996285 ("self-contained CRIU bundle via RPATH so cuda-checkpoint uses the container glibc") fixed a real bug in this area. Building from source let us control link-time glibc; a prebuilt has fixed expectations. Needs a smoke test against our actual workload images, since this class of failure passes unit tests and only aborts at restore.

3. Extensibility. Owning the source is the hook for gpuPairs GPU migration (r580+), which is the direction multi-GPU work in ci(gha): add Bazel build + test workflow #25 has been heading. Confirm the upstream binary exposes migration before giving up the hook.

Sequencing

Hold the removal until the in-flight manifest/chart cluster validation completes. Stacking a toolchain swap on top of unvalidated manifest changes makes any failure hard to attribute.

[balajinvda]

Direction change: keep nvsnap-cuda-checkpoint.c in-tree, but switch the build to ship NVIDIA's official binary permanently.

Rationale: retaining the source keeps the extensibility hook (r580 gpuPairs migration, lock timeouts) without carrying the maintenance of a binary we build ourselves, while the shipped artifact becomes the one NVIDIA tests and supports. Deleting the source buys nothing and loses the hook.

Rescoped work:

• docker/agent/Dockerfile.base: replace the cuda-cli-builder stage with the upstream binary (x86-64 from bin/, aarch64 from bin/aarch64_Linux).
• scripts/build-agent.sh (~148-170): drop the source-copy step.
• Keep docker/agent/nvsnap-cuda-checkpoint.c and note in its header that it is a reference/fallback, not the shipped artifact.
• docker/agent/cuda-checkpoint-wrapper.sh stays; it still points LD_LIBRARY_PATH at the node driver libcuda.

Still to check before the switch, unchanged from the original scope:

• glibc compatibility. 499628524 ("self-contained CRIU bundle via RPATH so cuda-checkpoint uses the container glibc") fixed a real bug here. Building from source let us control link-time glibc; a prebuilt has fixed expectations. Needs a smoke test against the actual workload images, since this failure mode passes unit tests and only shows up at restore.
• Driver floor. Upstream gates ARM CPU support at r595. Worth noting this is largely moot in practice: per the capture-path matrix, arm uses the rootfs path and does not invoke cuda-checkpoint at all. The only cell that uses it is x86 single-GPU, where the official x86 binary is the better-tested artifact.