All Apps and Add-ons

Applying conf updates to master & forwarders via Kubernetes

nickgleed
Explorer

Im new to both k8 and splunk. I'm interested in the best practice of deploying changes of various conf files to either search/indexers or to forwarders. My intention is to embed conf changes into my helm deployments and either spin up or update the pods as applicable via helm. Where forwarders are baked into an AMI for non dockerised apps the updates will scripted into a "golden image/AMI" via jenkins.

Is this approach OK - or should we apply conf updates via jenkins by uploading files onto the master and let splunk replicate the conf updates? I hope this makes sense.

Tags (1)
0 Karma

vliggio
Communicator

For indexers, if you're using a master server, you really should use the master to control your configs. How you control your master is up to you - if you want to use helm, jenkins, whatever, that's fine - just use the Splunk commands to push those configs to the indexers. This ensures that the configs are consistent and you can also control reboots somewhat more easily.

Search heads are similar - if you're using a search cluster, push your configs to the deployers using your tool of choice, and use the Splunk cli commands to push the configs out to the search cluster. Individual search heads can be configured any way you like.

I have mixed feelings about using configuration tools to manage forwarders. I've found it's a lot harder to ensure configurations if you have any questions about your config management. I've seen chef deployments where there were 100's of non-compliant forwarders because of chef issues, and if they were all pointing to a deployment server, you would then have a centralized place to look at what Splunk is up to. Now is the deployment server perfect? Far from it; it has a lot of shortcomings. But in general it's the easiest choice to start out with. You might want a base config that you push to the forwarders for initial installation and then push the rest from a deployment server. Be careful in that type of scenario - you can accidentally have competing configuration management systems.

Remember Splunk is mostly stateful (the master is completely stateless, but the other systems keep varying degrees of state) so things like containers, autoscaling groups in AWS, etc, where you swap in/out base systems is not really ideal with the current Splunk design. Maybe in the future, if they consider containerization in their design, then it will be easier to containerize.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...