nvsnap: criu-v2 restore depends on the target PID namespace happening to be free

[balajinvda]

Problem

criu-v2 restore rebuilds the process tree at the exact PIDs recorded in the
dump, using clone3 with set_tid. It restores into the placeholder pod's
existing PID namespace, so it needs every one of those PIDs to be unused. When
one is not, restore dies at the first collision:

Error (criu/cr-restore.c:1242): Can't fork for 336: File exists

Nothing guarantees they are free. The dump does not serialize a PID namespace,
which the restore log states plainly:

No pidns-1.img image

So correctness here rests on a coincidence: that whatever the placeholder and
the agent have allocated does not overlap the dumped tree's PID range.

Current mitigation

#893 pushes the placeholder's next PID allocation to 100000 before
CRIU runs (reservePlaceholderPIDs, internal/agent/restore_v2.go), which the
agent does from the host because a container cannot write
/proc/sys/kernel/ns_last_pid at all. That removes the realistic collision
source and NIM restores reliably with it.

It is still a mitigation. It narrows the window rather than closing it:

• It is best-effort. If the bump fails the restore proceeds unprotected, by
  design, so that a failure here cannot break restores that would have worked.
• It assumes no dumped PID exceeds the floor. True today, but it is an
  assumption about workload shape, not an invariant.
• Anything that allocates in that namespace between the bump and the clone3 can
  still take a PID.

Proposed fix

Dump and restore the PID namespace, so CRIU creates a fresh one and every PID
in it is available by construction. The exact-PID requirement stops being a
constraint on the environment and becomes an internal detail.

This needs design work rather than a flag:

• The placeholder currently provides the namespace CRIU restores into. With a
  restored PID namespace, what the placeholder is for, and what pid 1 becomes,
  both change.
• Kubernetes and containerd have their own view of the container's init
  process. A restored PID namespace has to stay consistent with what the
  runtime believes it is supervising, or lifecycle and reaping break.
• The cgroup placement work (clone3 CLONE_INTO_CGROUP, see restore_v2.go)
  interacts with this and should be revisited alongside it.

Why it matters beyond the current failure

The risk scales with how many processes a workload starts before it settles,
because that determines how high its session leader PID sits. NIM's entrypoint
chain puts it at 336; vLLM and SGLang land far lower. Larger multi-process
serving stacks will sit higher still, so this becomes more likely as workloads
get bigger, not less.

Acceptance criteria

• A criu-v2 restore succeeds without depending on the target namespace's PID
  allocation state.
• The ns_last_pid reservation can be removed, or is demonstrably redundant.
• A regression test restores a tree whose PIDs are deliberately occupied in the
  target namespace beforehand.

References

• Mitigation and measurements: fix(nvsnap): make TensorRT-LLM checkpointable, unblock NIM capture, and stop repeat captures from corrupting the container #893
• Related: nvsnap: second capture on the same pod fails - image harvest corrupts the container's overlay mount #892 (repeat capture corrupts the container's overlay)

[balajinvda]

Non-root images cannot self-reserve their pid range, which limits the fix that
just landed for the restore collisions.

Measured in the NIM image:

uid=1000(nim) gid=1000(nim)
echo 100000 > /proc/sys/kernel/ns_last_pid
  -> write error: Operation not permitted

privileged: true does not confer root, so the reservation the restore
placeholder performs is a no-op wherever the image's default user is not root.
The agent's new guard catches this and refuses the restore by name rather than
failing partway through with clone3 EEXIST, so it is loud rather than silent.

For our own test manifests this is fixable by running the placeholder as
runAsUser: 0 (done for nim-llama-8b). The placeholder only bumps and tails;
CRIU restores the workload with the uid recorded in the checkpoint, so the
workload's own user is unchanged.

For production it is a real constraint. Restore pods run the tenant's image, and
running non-root is normal practice, so reservePIDRange in restore-entrypoint
will be a no-op for a large share of real workloads. The options look like:

1. Have the webhook force runAsUser: 0 on restore pods. Changes the tenant's
   security posture, and is probably not acceptable on its own.
2. Reserve from the agent, on the host, where privileges are not in question.
   The difficulty is timing: the bump has to land before the pod starts anything
   long-lived, and the pod's own startup is not under our control.
3. Dump the pid namespace root so restore creates a fresh namespace. This removes
   the requirement rather than satisfying it, and is the only option that does
   not depend on the tenant's uid. See docs/proposals/pidns-capture.md.

Option 3 was previously treated as a cleanup. This finding makes it the
load-bearing one for production, since options 1 and 2 both depend on
circumstances we do not control.