Centrally controlling Configuration Chnages on indexers

[shocko]

I have a Splunk 9.0.4 estate on Windows 2019 with the following:

• Search head
• 2 x indexers
• Cluster master/deployment server

I'm trying to automate all deployments of apps to forwarders and all configuration on indexers (transforms/prop.conf) etc.  For the apps that go to the universal forwarders this has been straightforward and I simply add them to deployment server and they push out.

What I am not clear on is how I might manage pushing out configuration to my indexers in centralised controlled manner. For example, say I have an app that has a component that needs to be pushed to the forwarder to gather events but then a prop.conf modification to increase the TRUNCATE size. How can I do this centrally? 

PS: Apologies if this is somewhat of a noob questions! I'm a long term Splunk tinkerer but I only dip into it when my role necessitates it. 

[shocko]

Greta answer and thanks for taking the time. 

[isoutamo · Accepted solution]

Hi

Events DS can update those configurations to SH and CMI don't encourage you to use those for anything else than UFs. In some cases you could use those also for HFs, but e.g. modular inputs has many TAs which are not supported for DS.

How I have done this?

Usually I have own git repositories for all configurations. I just update conf files etc. on there and the commit & push those with pull request to main branch. Then I have some kind of CI/CD pipe (depends on case/customer) which generates ready to install app/ta-packages for SH, CM, DS etc. Then just install/deploy those packages to correct places. This can do manually or automatic depending on case/customer.

On individual SH this mean e.g. install via GUI or via command line "splunk install app <package name> [-update 1]). On SHC this must do via Deployer quite similar than CM but use etc/shcluster directory instead of manager-apps.

On indexer cluster this means, that I just untar those to correct place 

tar xvzf <package.spl> -C /opt/splunk/etc/manager-apps
splunk apply cluster bundle
splunk show cluster-bundle-status

Push indexers only those configurations which are necessary there like some props&transforms.conf. And create own apps for those, that will be easier for long run.

Depending on your environment you could automate everything after you have push your changes to repository and have done PR with merge. How those automation steps are done, is depending which tools you have on your environment.

r. Ismo