Deployment Architecture
Highlighted

Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

Explorer

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how?

And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly?
Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert?

Highlighted

Re: Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

Legend

Hi @st4ple.
why do you think that to deploy indexes.conf to Search Head Cluster members is a good practice?
It's correct to send all the logs to indexes, so on Search Heads there aren't indexes, still Summary indexes must be sent to Indexers clusters when you have a Search Head Cluster.
Infact you have an error on restarting.
Anyway, you have to manage SH members only by Deployer, so you can be sure about deployed configurations.
On SH you have to deploy only Apps without indexes.conf.

Ciao.
Giuseppe

0 Karma
Highlighted

Re: Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "orgallindexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "orgidxindexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, orgshindexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, an upvote would be appreciated.
0 Karma
Highlighted

Re: Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

Esteemed Legend

Yes, you should, for 2 main reasons:
1: If you need to use | collect or | mcollect, or the Log Event or Add to Summary Index Alert Actions, they will not work if the Search Head does not have an index definition in place:
2: The type-ahead completion feature will not work for index=....

In PS we have base config apps and one of them is the YourCompany_all_indexes app and this gets deployed to your Indexers and your Deployer, where you push it out to your SHC members.

0 Karma