You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Subject: Issue with Log Collection from Custom Path (/data/var/log/pods) #303
I am using coroot-node-agent for log collection in my Kubernetes cluster.
In our environment, the Kubelet log directory is changed to a custom path: /data/var/log/pods (instead of the default /var/log/pods).
Currently, it seems that the agent is not collecting any logs from this location, and there are no specific error messages regarding this.
Does coroot-node-agent automatically discover custom log paths?
If not, is there a configuration or flag to specify a custom log directory?
If this is not yet supported, could you please consider adding support for custom log paths?
I tried to reproduce your exact setup. Short version: pod stdout/stderr logs are collected from a custom podLogsDir automatically in the current code — the pod log directory is not hardcoded to /var/log/pods. So this should already work; if it doesn't for you, the cause is most likely one of the points at the bottom. Details + a full repro below.
How pod log collection works
For containers managed by containerd / CRI-O / Docker the agent does not assume /var/log/pods. It reads the log file path directly from the container-runtime metadata:
That absolute path (whatever kubelet told the runtime) is then tailed via /proc/1/root/<logPath> in runLogParser (containers/container.go). Because the DaemonSet runs with hostPID: true + privileged, /proc/1/root is the host root, so a custom path like /data/var/log/pods — even on a separate disk/mount — is reachable.
The host has no /var/log/pods at all, only the custom dir:
$ ls /var/log/podsls: cannot access '/var/log/pods': No such file or directory
$ ls /data/var/log/podsdefault_noisy_<uid> kube-system_... ...
And the agent actually tails the custom path:
$ kubectl -n coroot logs ds/coroot-node-agent | grep "container logparser"... "started container logparser" cg=".../cri-containerd-<id>.scope"
# the agent process holds an open fd on the custom path:
$ ls -l /proc/$(pgrep coroot-node-agent)/fd | grep /data/var/log/pods... 207 -> /data/var/log/pods/default_noisy_<uid>/noisy/0.log
I also verified this works when /data is a separate mount (a distinct filesystem, and even one mounted after the agent started): /proc/1/root resolves in the host mount namespace, so the separate volume is still traversed.
So if it isn't working for you, the usual suspects are:
The runtime isn't reporting a log path (empty Metadata.LogPath) → the agent silently skips it (containers/container.go: if c.metadata.logPath == "" { return }). Could you share crictl inspect <id> | grep logPath for one of your pods?
The agent isn't running with hostPID: true / privileged (custom Helm values?) → then /proc/1/root is the agent's own container root, not the host, and no host path is reachable. Could you share your DaemonSet spec (or at least hostPID/securityContext)?
Older agent version — which image tag are you on?
Also handy: kubectl -n <ns> logs ds/coroot-node-agent | grep -i logparser — do you see any started container logparser lines?
One caveat — in case you actually meant application log files
There is a second, separate mechanism for logs an application writes to files itself (not stdout/stderr). That one is hardcoded to /var/log/ (ebpftracer/init.go and resolveFd in containers/container.go), excluding /var/log/{pods,containers,journal}. Custom directories for app-written log files are not picked up today, and there's no flag for it. I confirmed this too: an app writing to /var/log/myapp/app.log is collected, while the same app writing to /srv/customlogs/app.log is silently ignored — the only difference being the path prefix. If that's your case rather than pod stdout, it's a different fix and worth a feature discussion.
Could you share the three things above (crictl logPath, DaemonSet hostPID/securityContext, agent version + any logparser log lines)? That will pin down which of the two cases you're hitting.
We are experiencing the same issue with coroot-node-agent:1.33.3.
In our environment, /var/log/pods is a symbolic link to a custom log directory:
/var/log/pods -> /data/var/log/pods
The symbolic link works correctly on the Kubernetes node, but coroot-node-agent does not collect logs through it. No relevant error message is shown in the agent logs.
It appears that the agent either does not follow symbolic links or cannot access the symlink target inside the container.
Could you confirm whether symbolic links for /var/log/pods are supported?
It would also be helpful if the agent supported one of the following:
Following symbolic links for the default /var/log/pods path
A configuration option or command-line flag for specifying a custom pod log directory
Activity
gecube commentedon Jul 6, 2026
Hi @3siksfather 👋
I tried to reproduce your exact setup. Short version: pod stdout/stderr logs are collected from a custom
podLogsDirautomatically in the current code — the pod log directory is not hardcoded to/var/log/pods. So this should already work; if it doesn't for you, the cause is most likely one of the points at the bottom. Details + a full repro below.How pod log collection works
For containers managed by containerd / CRI-O / Docker the agent does not assume
/var/log/pods. It reads the log file path directly from the container-runtime metadata:Metadata.LogPath(containers/containerd.go)LogPath(containers/crio.go)LogPath(containers/dockerd.go)That absolute path (whatever kubelet told the runtime) is then tailed via
/proc/1/root/<logPath>inrunLogParser(containers/container.go). Because the DaemonSet runs withhostPID: true+privileged,/proc/1/rootis the host root, so a custom path like/data/var/log/pods— even on a separate disk/mount — is reachable.Reproduction (kind, k8s v1.36)
kind.yamlwith a custom kubelet pod-log dir:kind create cluster --config kind.yaml kubectl run noisy --image=busybox -- sh -c 'i=0; while true; do echo hi-$i; i=$((i+1)); sleep 1; done' kubectl apply -f manifests/coroot-node-agent.yamlThe runtime reports the custom path:
The host has no
/var/log/podsat all, only the custom dir:And the agent actually tails the custom path:
I also verified this works when
/datais a separate mount (a distinct filesystem, and even one mounted after the agent started):/proc/1/rootresolves in the host mount namespace, so the separate volume is still traversed.So if it isn't working for you, the usual suspects are:
Metadata.LogPath) → the agent silently skips it (containers/container.go:if c.metadata.logPath == "" { return }). Could you sharecrictl inspect <id> | grep logPathfor one of your pods?hostPID: true/privileged(custom Helm values?) → then/proc/1/rootis the agent's own container root, not the host, and no host path is reachable. Could you share your DaemonSet spec (or at leasthostPID/securityContext)?Also handy:
kubectl -n <ns> logs ds/coroot-node-agent | grep -i logparser— do you see anystarted container logparserlines?One caveat — in case you actually meant application log files
There is a second, separate mechanism for logs an application writes to files itself (not stdout/stderr). That one is hardcoded to
/var/log/(ebpftracer/init.goandresolveFdincontainers/container.go), excluding/var/log/{pods,containers,journal}. Custom directories for app-written log files are not picked up today, and there's no flag for it. I confirmed this too: an app writing to/var/log/myapp/app.logis collected, while the same app writing to/srv/customlogs/app.logis silently ignored — the only difference being the path prefix. If that's your case rather than pod stdout, it's a different fix and worth a feature discussion.Could you share the three things above (crictl
logPath, DaemonSethostPID/securityContext, agent version + anylogparserlog lines)? That will pin down which of the two cases you're hitting.3siksfather commentedon Jul 31, 2026
Hi,
We are experiencing the same issue with
coroot-node-agent:1.33.3.In our environment,
/var/log/podsis a symbolic link to a custom log directory:The symbolic link works correctly on the Kubernetes node, but
coroot-node-agentdoes not collect logs through it. No relevant error message is shown in the agent logs.It appears that the agent either does not follow symbolic links or cannot access the symlink target inside the container.
Could you confirm whether symbolic links for
/var/log/podsare supported?It would also be helpful if the agent supported one of the following:
/var/log/podspathEnvironment:
gecube commentedon Jul 31, 2026
@3siksfather Hi! Thanks for the new data. I wanna to ask you just one crucial thing. You mentioned two path:
Are both located on the same volume? Or maybe do you use a dedicated volume for
/data?3siksfather commentedon Aug 4, 2026
@gecube dedicated volume /dev/sdb /data
sudo ln -s /data/var/log/pods /var/log/pods