nvsnap: NIM checkpoint fails on /dev/gdrdrv file descriptor (gdrcopy)

[balajinvda]

Summary

NIM workloads cannot be checkpointed with CRIU + cuda-checkpoint. The NIM server
process holds an open file descriptor on /dev/gdrdrv (the GPUDirect RDMA /
gdrcopy driver), and CRIU has no handler for a file descriptor on that device,
so the dump aborts.

This is specific to NIM: vLLM, SGLang and TensorRT-LLM do not open gdrdrv, which
is why the issue has not surfaced on the other engines.

Symptom

Error (criu/files-ext.c:98): Can't dump file 36 of that type [20666] (chr 506:0)
Error (criu/cr-dump.c:1737): Dump files (pid: 78) failed with -1
Error (criu/cr-dump.c:2170): Dumping FAILED.

Major 506 is gdrdrv (dynamically allocated; the number varies per host, the
name does not). Confirmed from inside the workload container:

/proc/78/fd/36 -> /dev/gdrdrv
crw-rw-rw- 1 nim nim 506, 0 /dev/gdrdrv

Everything before this point works: the pod starts, serves inference, CRIU
seizes all four tasks, and cuda_plugin runs and resolves GPU state. The dump
fails only when collecting this one fd, roughly 110s in.

Why the obvious fix does not work

/dev/nvidia* file descriptors survive because cuda_plugin registers real
dump/restore handlers for them, not because of the --external dev[maj/min]
entries the agent passes.

Declaring gdrdrv the same way was tried and does not help. The agent now emits
--external dev[506/0]:gdrdrv, verified reaching CRIU (externals=13 = 12
nvidia char devices plus gdrdrv), and the dump fails identically. dev[]
externals do not apply to open file descriptors.

Direction

The candidate mechanism is CRIU's --external file[mnt_id:inode] form, which
tells CRIU to skip an fd at dump and reopen it by path at restore. This needs
confirming against CRIU's external.c / files-ext.c before implementing —
whether that form covers character-device fds is exactly the assumption that
was wrong the first time.

Worth evaluating alongside:

• Whether gdrcopy state is meaningful to preserve at all, or whether reopening
  a fresh fd at restore is sufficient. If gdrdrv holds pinned-memory
  registrations, reopening may not be enough and this becomes a plugin-level
  problem rather than an external-declaration one.
• Whether NIM can be configured to not use gdrcopy. That would be a workaround
  rather than a fix, and it changes what the test actually exercises.
• Routing NIM to the rootfs capture path, which does not use CRIU. Also a
  workaround, and it gives up live process state.

State of the workload

The manifest work is done and unrelated to this blocker. nim-llama-8b is
migrated to the criu-v2 convention (no interception stack, no
restore-entrypoint placeholder, workload launched under setsid, stdio to a
writable path since the image runs non-root). It starts, serves, and seizes
cleanly. This fd is the only remaining failure.

Note for whoever picks this up

The ["nvidia*", "gdrdrv"] glob in nvidiaDevExternals is present but does
not fix this. It should not be read as gdrdrv being handled.