splunk-connect-for-kubernetes - /var/log/containers/ log unreadable. It is excluded and would be examined next time.

[xindeNokia]

After deploy splunk-connect-for-k8s 1.3, I saw lots of warning msg from splunk-splunk-kubernetes-logging:

2020-02-19 02:50:09 +0000 [warn]: #0 [containers.log] /var/log/containers/splunk-splunk-kubernetes-logging-4wqnj_default_splunk-fluentd-k8s-logs-e403ac1d989b252566536f844f3817ac9334ac1dbc80bbfeda04fb063dac65a8.log unreadable. It is excluded and would be examined next time.

it seems all of the pods logs are skipped and unreadable, I can find them under /var/log/containers, but they are softlinks.
Not sure if this caused any issues. K8s cluster is deployed by kubespray.

lrwxrwxrwx 1 root root 127 Feb 18 21:47 splunk-splunk-kubernetes-logging-4wqnj_default_splunk-fluentd-k8s-logs-e403ac1d989b252566536f844f3817ac9334ac1dbc80bbfeda04fb063dac65a8.log -> /var/log/pods/default_splunk-splunk-kubernetes-logging-4wqnj_56c07fb2-d86d-411b-92a4-7c214919a33d/splunk-fluentd-k8s-logs/0.log

Wonder if anyone has seen this before and maybe I misconfigured something or there is any solutions for this...

Thanks in advance!

[xindeNokia · Accepted solution]

Found the solution for my question -

./splunk-connect-for-kubernetes/charts/splunk-kubernetes-logging/values.yaml: path: /var/log/containers/.log
Changed to:
path: /var/log/pods/.log works to me.