Deployment Architecture

Deploy indexes.conf in a Search Head Cluster? How to avoid (and recover in case of) misconfiguration?

st4ple
Path Finder

We have a Search Head Cluster connected to an Indexer Cluster. All indexes are on the clustered Indexers, and the Search Head Cluster members forward their local internal indexes to the Indexers. Is it best practice to still deploy a copy of the "master" indexes.conf (that gets distributed to the Indexers through the Cluster Master) to the Search Head Cluster members? If so, how?

And much more importantly: How do we recover from misconfigurations that stop the Search Head Cluster members from restarting correctly?
Scenario: we use the Deployer to deploy a version of indexes.conf that contains a reference to a volume e.g. in homePath that is not defined on the Search Head Cluster members. The Search Head Cluster members will initiate a rolling-restart but not come back online as Splunkd will notice that there are incorrectly defined indexes on the instance. How can we a) avoid this happening and b) if it happens, quickly revert?

1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

woodcock
Esteemed Legend

Yes, you should, for 2 main reasons:
1: If you need to use | collect or | mcollect, or the Log Event or Add to Summary Index Alert Actions, they will not work if the Search Head does not have an index definition in place:
2: The type-ahead completion feature will not work for index=....

In PS we have base config apps and one of them is the YourCompany_all_indexes app and this gets deployed to your Indexers and your Deployer, where you push it out to your SHC members.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, you should have your indexes defined on your SHC. This allows for auto-complete of index names and populates dropdowns in many UI elements.

The secret is to use apps. Three of them. The main app, which I'll call "org_all_indexes", contains indexes.conf with the index definitions, but not the volume definitions. App 2, "org_idx_indexes" contains an indexes.conf with only the volume definitions used by indexers. App 3, org_sh_indexes, has an indexes.conf file with 'fake' volume definitions to prevent the errors you've seen.

Deploy apps 1 and 2 to your indexers via master-apps. Deploy apps 1 and 3 to the SHC using your deployer.

---
If this reply helps you, Karma would be appreciated.

gcusello
SplunkTrust
SplunkTrust

Hi @st4ple.
why do you think that to deploy indexes.conf to Search Head Cluster members is a good practice?
It's correct to send all the logs to indexes, so on Search Heads there aren't indexes, still Summary indexes must be sent to Indexers clusters when you have a Search Head Cluster.
Infact you have an error on restarting.
Anyway, you have to manage SH members only by Deployer, so you can be sure about deployed configurations.
On SH you have to deploy only Apps without indexes.conf.

Ciao.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...