Deployment Architecture

Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

st4ple
Path Finder

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how?

And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly?
Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert?

1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

Yes, you should, for 2 main reasons:
1: If you need to use | collect or | mcollect, or the Log Event or Add to Summary Index Alert Actions, they will not work if the Search Head does not have an index definition in place:
2: The type-ahead completion feature will not work for index=....

In PS we have base config apps and one of them is the YourCompany_all_indexes app and this gets deployed to your Indexers and your Deployer, where you push it out to your SHC members.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

gcusello
SplunkTrust
SplunkTrust

Hi @st4ple.
why do you think that to deploy indexes.conf to Search Head Cluster members is a good practice?
It's correct to send all the logs to indexes, so on Search Heads there aren't indexes, still Summary indexes must be sent to Indexers clusters when you have a Search Head Cluster.
Infact you have an error on restarting.
Anyway, you have to manage SH members only by Deployer, so you can be sure about deployed configurations.
On SH you have to deploy only Apps without indexes.conf.

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...