Skip to content

Subject: Issue with Log Collection from Custom Path (/data/var/log/pods) #303

Open

Description

@3siksfather

Content:
Hi,

I am using coroot-node-agent for log collection in my Kubernetes cluster.
In our environment, the Kubelet log directory is changed to a custom path: /data/var/log/pods (instead of the default /var/log/pods).

Currently, it seems that the agent is not collecting any logs from this location, and there are no specific error messages regarding this.

Does coroot-node-agent automatically discover custom log paths?

If not, is there a configuration or flag to specify a custom log directory?

If this is not yet supported, could you please consider adding support for custom log paths?

Thanks!

Activity

  1. gecube commented on Jul 6, 2026

    @gecube

    Hi @3siksfather 👋

    I tried to reproduce your exact setup. Short version: pod stdout/stderr logs are collected from a custom podLogsDir automatically in the current code — the pod log directory is not hardcoded to /var/log/pods. So this should already work; if it doesn't for you, the cause is most likely one of the points at the bottom. Details + a full repro below.

    How pod log collection works

    For containers managed by containerd / CRI-O / Docker the agent does not assume /var/log/pods. It reads the log file path directly from the container-runtime metadata:

    • containerd — Metadata.LogPath (containers/containerd.go)
    • CRI-O — LogPath (containers/crio.go)
    • Docker — LogPath (containers/dockerd.go)

    That absolute path (whatever kubelet told the runtime) is then tailed via /proc/1/root/<logPath> in runLogParser (containers/container.go). Because the DaemonSet runs with hostPID: true + privileged, /proc/1/root is the host root, so a custom path like /data/var/log/pods — even on a separate disk/mount — is reachable.

    Reproduction (kind, k8s v1.36)

    kind.yaml with a custom kubelet pod-log dir:

    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    nodes:
      - role: control-plane
        kubeadmConfigPatches:
          - |
            kind: KubeletConfiguration
            podLogsDir: /data/var/log/pods
    kind create cluster --config kind.yaml
    kubectl run noisy --image=busybox -- sh -c 'i=0; while true; do echo hi-$i; i=$((i+1)); sleep 1; done'
    kubectl apply -f manifests/coroot-node-agent.yaml

    The runtime reports the custom path:

    $ crictl inspect <container-id> | grep logPath
    "logPath": "/data/var/log/pods/default_noisy_<uid>/noisy/0.log"

    The host has no /var/log/pods at all, only the custom dir:

    $ ls /var/log/pods
    ls: cannot access '/var/log/pods': No such file or directory
    $ ls /data/var/log/pods
    default_noisy_<uid>  kube-system_...  ...

    And the agent actually tails the custom path:

    $ kubectl -n coroot logs ds/coroot-node-agent | grep "container logparser"
    ... "started container logparser" cg=".../cri-containerd-<id>.scope"
    
    # the agent process holds an open fd on the custom path:
    $ ls -l /proc/$(pgrep coroot-node-agent)/fd | grep /data/var/log/pods
    ... 207 -> /data/var/log/pods/default_noisy_<uid>/noisy/0.log

    I also verified this works when /data is a separate mount (a distinct filesystem, and even one mounted after the agent started): /proc/1/root resolves in the host mount namespace, so the separate volume is still traversed.

    So if it isn't working for you, the usual suspects are:

    1. The runtime isn't reporting a log path (empty Metadata.LogPath) → the agent silently skips it (containers/container.go: if c.metadata.logPath == "" { return }). Could you share crictl inspect <id> | grep logPath for one of your pods?
    2. The agent isn't running with hostPID: true / privileged (custom Helm values?) → then /proc/1/root is the agent's own container root, not the host, and no host path is reachable. Could you share your DaemonSet spec (or at least hostPID/securityContext)?
    3. Older agent version — which image tag are you on?

    Also handy: kubectl -n <ns> logs ds/coroot-node-agent | grep -i logparser — do you see any started container logparser lines?

    One caveat — in case you actually meant application log files

    There is a second, separate mechanism for logs an application writes to files itself (not stdout/stderr). That one is hardcoded to /var/log/ (ebpftracer/init.go and resolveFd in containers/container.go), excluding /var/log/{pods,containers,journal}. Custom directories for app-written log files are not picked up today, and there's no flag for it. I confirmed this too: an app writing to /var/log/myapp/app.log is collected, while the same app writing to /srv/customlogs/app.log is silently ignored — the only difference being the path prefix. If that's your case rather than pod stdout, it's a different fix and worth a feature discussion.

    Could you share the three things above (crictl logPath, DaemonSet hostPID/securityContext, agent version + any logparser log lines)? That will pin down which of the two cases you're hitting.

  2. 3siksfather commented on Jul 31, 2026

    @3siksfather
    Author

    Hi,

    We are experiencing the same issue with coroot-node-agent:1.33.3.

    In our environment, /var/log/pods is a symbolic link to a custom log directory:

    /var/log/pods -> /data/var/log/pods
    

    The symbolic link works correctly on the Kubernetes node, but coroot-node-agent does not collect logs through it. No relevant error message is shown in the agent logs.

    It appears that the agent either does not follow symbolic links or cannot access the symlink target inside the container.

    Could you confirm whether symbolic links for /var/log/pods are supported?

    It would also be helpful if the agent supported one of the following:

    • Following symbolic links for the default /var/log/pods path
    • A configuration option or command-line flag for specifying a custom pod log directory
    • Mounting or scanning additional pod log paths

    Environment:

    coroot-node-agent version: 1.33.3
    Default path: /var/log/pods
    Symlink target: /data/var/log/pods
    
  3. gecube commented on Jul 31, 2026

    @gecube

    @3siksfather Hi! Thanks for the new data. I wanna to ask you just one crucial thing. You mentioned two path:

    /var/log/pods -> /data/var/log/pods
    

    Are both located on the same volume? Or maybe do you use a dedicated volume for /data ?

  4. 3siksfather commented on Aug 4, 2026

    @3siksfather
    Author

    @gecube dedicated volume /dev/sdb /data
    sudo ln -s /data/var/log/pods /var/log/pods

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions