Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

[st4ple]

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how?

And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly?
Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert?

[woodcock]

Yes, you should, for 2 main reasons:
1: If you need to use | collect or | mcollect, or the Log Event or Add to Summary Index Alert Actions, they will not work if the Search Head does not have an index definition in place:
2: The type-ahead completion feature will not work for index=....

In PS we have base config apps and one of them is the YourCompany_all_indexes app and this gets deployed to your Indexers and your Deployer, where you push it out to your SHC members.

[richgalloway · Accepted solution]

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

[gcusello]

Hi @st4ple.
why do you think that to deploy indexes.conf to Search Head Cluster members is a good practice?
It's correct to send all the logs to indexes, so on Search Heads there aren't indexes, still Summary indexes must be sent to Indexers clusters when you have a Search Head Cluster.
Infact you have an error on restarting.
Anyway, you have to manage SH members only by Deployer, so you can be sure about deployed configurations.
On SH you have to deploy only Apps without indexes.conf.

Ciao.
Giuseppe