kubernetes 1.9.4 breaking changes: Universal Forwarder

[gcyre]

I've setup splunk universal forwarder as a daemonset on our kubernetes cluster. 2 nodes are running kuberntes 1.9.3 and one is running 1.9.4. On the 1.9.4 node the splunk forwarder pod is unable to start:

chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system

I believe this is related to changes recently made in 1.9.4:
https://github.com/kubernetes/kubernetes/pull/58720

Wondering if anyone has come across this or has a workaround?

thanks
Garry

[sechitwood]

Here is an issue with a the right way to mount configmaps.

Mount to /var/opt/splunk/etc and the entrypoint will copy to the right place. Posting this here since it still shows up as the top search result.

https://github.com/splunk/docker-splunk/issues/70

[ungborib]

I opened the github issue: https://github.com/splunk/docker-splunk/issues/70

[guilmxm]

Hello,

Would mind sharing your yaml daemonset configuration, I would be interested in troubleshooting this.
Have you tried running the ds without a persistent storage if you have one currently ?

Regards,

Guilhem

[ungborib]

sure. here it is: https://gist.github.com/erks/6e0d761cb27a0fec60f13beecbba670f#file-splunk-yml

[guilmxm]

Many thanks, will have a look 😉

[guilmxm]

@ungborib: I just noticed a reply in your Git issue, and indeed using "/var/opt/splunk" within the volume mount directive fixed the read only issue (just tested in a 1.10 cluster)

[peter7431]

I got similar messages when using a configmap and mounting it to /opt/splunk/etc/system/local in my pods. I ended up adding my deployment.conf file to the container, and making deployment apps for the rest of the config.