Is it OK to roll out the same indexes.conf on all indexer peers via a configuration management tool rather than the indexer master?

[danielwan]

I am managing a Splunk indexer cluster. I understand the office approach to creating a replicable index is creating an indexes.conf on master than apply the bundle to peers, like the following articles have described.

https://answers.splunk.com/answers/218464/how-to-create-a-new-index-in-index-cluster-622.html
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Configurethepeerindexes
http://docs.splunk.com/Documentation/Splunk/6.2.1/Indexer/Updatepeerconfigurations#Distribute_the_co...

My situation is I use a configuration management tool e.g. Chef, to administrate the Splunk indexer cluster.

My questions are
a) is it OK to roll out the same indexes.conf to all indexer peers via configuration management tool rather than indexer master?
b) It seems that indexes.conf pushed to peers from master is not stored in /opt/splunk/etc/system/local/indexes.conf on peers. Any idea on where the change is stored?

[gjanders · Accepted solution]

I don't agree with mayurr98's comment here, in regard to:

a) is it OK to roll out the same
indexes.conf to all indexer peers via
configuration management tool rather
than indexer master?

I would say no, the cluster master should be the place you configure the bundle from, if you refer to How indexer cluster nodes start up when a peer joins or it's going to download the current bundle.
Also the master can validate if a bundle will trigger a restart or just require a reload.

b) It seems that indexes.conf pushed
to peers from master is not stored in
/opt/splunk/etc/system/local/indexes.conf
on peers. Any idea on where the change
is stored?

As per mayurr98 it will go into $SPLUNK_HOME/etc/slave-apps/

[mayurr98]

yes so if he wants to use configuration management tool then he needs to be specific about the what is the process after pushing a configuration. I gave an answer based on my personal experience.I have seen pushing configurations through hp tools.Eventually, it will be complex but it is doable. So it is always a best practice to use cluster master.

[gjanders]

Definitely an interesting perspective, how do you handle when the indexer requires a restart vs when it does not?

The cluster master would handle that for you which is why I suggested it wouldn't just be bad practice, it might not work as expected...(unless of course your reloading or restarting when you change config)

[mayurr98]

hey, these are answers to your questions:
a) Yes, it OK to roll out the same indexes.conf to all indexer peers via configuration management tool rather than indexer master. but it is best practice to do it from cluster master.
b) when you push any configuration from the master, it is getting stored in $SPLUNK_HOME/etc/slave-apps/ on peers.

Let me know if it helps!