I've setup splunk universal forwarder as a daemonset on our kubernetes cluster. 2 nodes are running kuberntes 1.9.3 and one is running 1.9.4. On the 1.9.4 node the splunk forwarder pod is unable to start:
chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system
I believe this is related to changes recently made in 1.9.4:
https://github.com/kubernetes/kubernetes/pull/58720
Wondering if anyone has come across this or has a workaround?
thanks
Garry
Here is an issue with a the right way to mount configmaps.
Mount to /var/opt/splunk/etc and the entrypoint will copy to the right place. Posting this here since it still shows up as the top search result.
I opened the github issue: https://github.com/splunk/docker-splunk/issues/70
Hello,
Would mind sharing your yaml daemonset configuration, I would be interested in troubleshooting this.
Have you tried running the ds without a persistent storage if you have one currently ?
Regards,
Guilhem
Many thanks, will have a look 😉
@ungborib: I just noticed a reply in your Git issue, and indeed using "/var/opt/splunk" within the volume mount directive fixed the read only issue (just tested in a 1.10 cluster)
I got similar messages when using a configmap and mounting it to /opt/splunk/etc/system/local in my pods. I ended up adding my deployment.conf file to the container, and making deployment apps for the rest of the config.