Getting Data In

kubernetes 1.9.4 breaking changes: Universal Forwarder

gcyre
New Member

I've setup splunk universal forwarder as a daemonset on our kubernetes cluster. 2 nodes are running kuberntes 1.9.3 and one is running 1.9.4. On the 1.9.4 node the splunk forwarder pod is unable to start:

chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system
chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system

I believe this is related to changes recently made in 1.9.4:
https://github.com/kubernetes/kubernetes/pull/58720

Wondering if anyone has come across this or has a workaround?

thanks
Garry

0 Karma

sechitwood
Engager

Here is an issue with a the right way to mount configmaps.

Mount to /var/opt/splunk/etc and the entrypoint will copy to the right place. Posting this here since it still shows up as the top search result.

https://github.com/splunk/docker-splunk/issues/70

ungborib
New Member
0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hello,

Would mind sharing your yaml daemonset configuration, I would be interested in troubleshooting this.
Have you tried running the ds without a persistent storage if you have one currently ?

Regards,

Guilhem

0 Karma

ungborib
New Member
0 Karma

guilmxm
SplunkTrust
SplunkTrust

Many thanks, will have a look 😉

0 Karma

guilmxm
SplunkTrust
SplunkTrust

@ungborib: I just noticed a reply in your Git issue, and indeed using "/var/opt/splunk" within the volume mount directive fixed the read only issue (just tested in a 1.10 cluster)

0 Karma

peter7431
Explorer

I got similar messages when using a configmap and mounting it to /opt/splunk/etc/system/local in my pods. I ended up adding my deployment.conf file to the container, and making deployment apps for the rest of the config.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...