debug: chroot fails for distroless images (eg. on minikube)

[ernstvonoelsen]

How to reproduce

1. K8s cluster installed/launched via minikube start --driver=docker ✔
2. mint installed from latest release:

    $ mint --version                                                                                                         
    mint version linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x
   

3. try to attach a sidecar container to the pod storage-provisioner in namespace kube-system using busybox:latest

Output

$ mint
>>> debug --runtime k8s --namespace kube-system --pod storage-provisioner --target storage-provisioner --debug-image busybox:latest
cmd=debug state=started

cmd=debug info=cmd.input.params namespace='kube-system' pod='storage-provisioner' runtime='k8s' target='storage-provisioner' terminal='true' gid='-1' uid='-1' fallback-to-target-user='true' debug-image='busybox:latest' entrypoint='[]' cmd='[]' run-as-target-shell='true' 
cmd=debug state=kubernetes.runtime.handler.started
cmd=debug info=wait.for.debug.container namespace='kube-system' name='mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64' pod='storage-provisioner' 
cmd=debug info=wait.for.container namespace='kube-system' type='ephemeral' id='docker://b5469506fae1ff94e74bbab33460e096222918192879c38ae43b3946a8d2b8f3' state='TERMINATED' exit_code='127' reason='Error' name='mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64' pod='storage-provisioner' 
time="2025-01-28T11:22:18+01:00" level=error msg=waitForContainer app=mint cmd=debug debug.container.name=mint-debugger-142661d818b412e5c564825d59bf81b2f5e94b64 error="Container terminated" op=debug.HandleKubernetesRuntime sid=142661d818b412e5c564825d59bf81b2f5e94b64
cmd=debug error=debug.container.error message='terminated'
cmd=debug info=container.logs.start                                                                                                                          
cmd=debug log='debug.container.logs' event=LOG.START  ====================
chroot: can't execute 'sh': No such file or directory

cmd=debug log='debug.container.logs' event=LOG.END  ====================
cmd=debug info=container.logs.end
cmd=debug state=debug.container.error code=-1
cmd=debug info=report file='slim.report.json'                                                                                                                
cmd=debug info=exit location='/usr/local/bin' code='-1' version='linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x' 
app='mint' message='GitHub Discussions' info='https://github.com/mintoolkit/mint/discussions'
app='mint' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S'       
app='mint' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/fAvq4ruKsG'   

$ cat slim.report.json
{
  "document": "doc.report.command",
  "version": "ov/command/debug/1.0",
  "engine": "linux/amd64|ALP|x.1.42.2|29e62e7836de7b1004607c51c502537ffe1969f0|2025-01-16_07:48:54AM|x",
  "containerized": false,
  "host_distro": {
    "name": "Ubuntu",
    "version": "22.04",
    "display_name": "Ubuntu 22.04.5 LTS"
  },
  "type": "debug",
  "state": "error"
}

Further info

I tried to manually repeat the steps as described here, i.e.

1. run kubectl debug -it -n kube-system storage-provisioner -t storage-provisioner --image busybox --profile="sysadmin"
2. run ln -s /proc/$$/root /proc/1/.sidecar -> first failure, because /proc/1 is read-only; ln -s /proc/$$/root /proc/1/root/.sidecar worked
3. run chroot /proc/1/root -> fails with /bin/sh: no such file or directory (chroot /proc/1/root .sidecar/bin/sh failed also)
4. check content of /proc/1/root/:

   / # ls -l /proc/1/root/
   total 30736
   drwxr-xr-x    5 root     root           360 Jan 28 10:22 dev
   drwxr-xr-x    2 root     root          4096 Jan 28 10:22 etc
   dr-xr-xr-x  505 root     root             0 Jan 28 10:22 proc
   -rwxr-xr-x    1 root     root      31465472 Mar 29  2021 storage-provisioner
   dr-xr-xr-x   13 root     root             0 Jan 28 10:21 sys
   drwxrwxrwt    5 root     root           140 Jan 28 10:29 tmp
   drwxr-xr-x    3 root     root          4096 Jan 28 10:22 var
   

5. create symlinks for each missing directory directly:

   # ln -s /proc/$$/root/bin /proc/1/root/bin
   # ln -s /proc/$$/root/usr /proc/1/root/usr
   

6. now chroot /proc/1/root works fine ✔

I do not know whether the root cause is the read-only file system or just the missing /bin, /usr directories themselves.
However, I think the workaround to iteratively link missing directories into the target fs could be implemented as a fallback.

Update

I found another way to directly chroot into the main process' root filesystem:

$ ln -s /proc/$$/root /proc/1/root/.sidecar
$ export PATH=$PATH:/.sidecar/bin:/.sidecar/usr/bin
$ chroot /proc/1/root /.sidecar/lib/ld-linux-x86-64.so.2 --library-path /.sidecar/lib /.sidecar/bin/sh

But now a simple ls command inside the new root must be executed as follows:

/.sidecar/lib/ld-linux-x86-64.so.2 --library-path /.sidecar/lib /.sidecar/bin/ls

[kcq]

@ernstvonoelsen looking into this... trying to repro. Thank you for opening the ticket! Not sure if it's the problem here, but I noticed that you are not using the preconfigured busybox debug image option. The latest version of busybox no longer statically compiles its binaries and it has an unfortunate side effect that might be showing up here. This is why the preconfigured busybox image in Mint is busybox@sha256:05a79c7279f71f86a2a0d05eb72fcb56ea36139150f0a75cd87e80a4272e4e39

[ernstvonoelsen]

Hi @kcq thanks for your feedback 👍

After reading about this project, I tried mint debug with busybox:musl, and it worked like a charm!

This is an excerpt from docker inspect busybox:musl:

"Id": "sha256:1aa9359d68a1730224b788b32440f488df351371b792ad83ea575e9c9c1f6dde",
"RepoDigests": [
            "busybox@sha256:37ccc05112d0a7162b605c375f5c0f5cfbc6e6c4fc8030ead6ae26a18c28a542"
        ]

[kcq]

Cdebug is a great dedicated tool! By the way, Ivan and I are doing a "Good Dockerfiles" campaign. It's not container debugging related though. If you've seen Dockerfiles that you feel can be improved or, the opposite, if you've seen Dockerfiles that you think are great examples of what Dockerfiles should look like please share those with us.

P.S.
Still investigating...

[ernstvonoelsen]

I watched your video with Felipe and Ivan, very interesting stuff 👍