topic Re: kubernetes 1.9.4 breaking changes: Universal Forwarder in Getting Data In

kubernetes 1.9.4 breaking changes: Universal Forwarder

gcyre — Fri, 16 Mar 2018 14:04:11 GMT

I've setup splunk universal forwarder as a daemonset on our kubernetes cluster. 2 nodes are running kuberntes 1.9.3 and one is running 1.9.4. On the 1.9.4 node the splunk forwarder pod is unable to start:
chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system
I believe this is related to changes recently made in 1.9.4:
https://github.com/kubernetes/kubernetes/pull/58720

Wondering if anyone has come across this or has a workaround?

thanks
Garry

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

peter7431 — Thu, 29 Mar 2018 21:18:45 GMT

I got similar messages when using a configmap and mounting it to /opt/splunk/etc/system/local in my pods. I ended up adding my deployment.conf file to the container, and making deployment apps for the rest of the config.

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

ungborib — Mon, 16 Apr 2018 19:59:00 GMT

I opened the github issue: https://github.com/splunk/docker-splunk/issues/70

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

guilmxm — Wed, 18 Apr 2018 21:07:59 GMT

Hello,

Would mind sharing your yaml daemonset configuration, I would be interested in troubleshooting this.
Have you tried running the ds without a persistent storage if you have one currently ?

Regards,

Guilhem

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

ungborib — Wed, 18 Apr 2018 21:35:40 GMT

sure. here it is: https://gist.github.com/erks/6e0d761cb27a0fec60f13beecbba670f#file-splunk-yml

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

guilmxm — Wed, 18 Apr 2018 23:01:09 GMT

Many thanks, will have a look 😉

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

guilmxm — Tue, 01 May 2018 22:05:40 GMT

@ungborib: I just noticed a reply in your Git issue, and indeed using "/var/opt/splunk" within the volume mount directive fixed the read only issue (just tested in a 1.10 cluster)

Re: kubernetes 1.9.4 breaking changes: Universal Forwarder

sechitwood — Mon, 06 Aug 2018 18:16:49 GMT

Here is an issue with a the right way to mount configmaps.

Mount to /var/opt/splunk/etc and the entrypoint will copy to the right place. Posting this here since it still shows up as the top search result.

https://github.com/splunk/docker-splunk/issues/70