Deployment Architecture

Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

st4ple
Path Finder

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how?

And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly?
Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert?

1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

Yes, you should, for 2 main reasons:
1: If you need to use | collect or | mcollect, or the Log Event or Add to Summary Index Alert Actions, they will not work if the Search Head does not have an index definition in place:
2: The type-ahead completion feature will not work for index=....

In PS we have base config apps and one of them is the YourCompany_all_indexes app and this gets deployed to your Indexers and your Deployer, where you push it out to your SHC members.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

gcusello
SplunkTrust
SplunkTrust

Hi @st4ple.
why do you think that to deploy indexes.conf to Search Head Cluster members is a good practice?
It's correct to send all the logs to indexes, so on Search Heads there aren't indexes, still Summary indexes must be sent to Indexers clusters when you have a Search Head Cluster.
Infact you have an error on restarting.
Anyway, you have to manage SH members only by Deployer, so you can be sure about deployed configurations.
On SH you have to deploy only Apps without indexes.conf.

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...