Performance suggestions for indexing cluster and search cluster.

[Cuyose]

We will be deploying a search head cluster to go along with out 10 indexer cluster. As it stands now these indexers are part of distributed search as search peers. We are having issues with many dashboards "unable to create search" which can be remedied by upping from the default max_base_searches in the limits.conf.

The issue is, our deployment is in AWS with indexers disk optimized instances and search heads cpu optimized. The indexers are already running very high CPU due to search activity while our search heads are idling at around 5-10% CPU.

When we deploy the search head cluster, should we be disabling the distributed search? It seems to be negatively impacting performance for us to have searches running on indexing nodes.

[lfedak_splunk]

Hey @Cuyose, if they solved your problem, remember to "√Accept" an answer to award karma points 🙂

[Steve_G_]

When you say that "these indexers are part of distributed search as search peers," do you mean that you connected them to the search heads through distsearch.conf, using one of the methods described here: http://docs.splunk.com/Documentation/Splunk/7.0.0/DistSearch/Configuredistributedsearch? If so, that's an incorrect configuration.

With indexer clustering, the indexers are automatically connected to the search heads when you add the search heads to the indexer cluster. Do not separately connect the indexers via distsearch.conf.

To address another point in your question, searches always run on indexers. The indexers hold the index data, and so the searches take place on the indexers. The search heads merely manage the overall search process. This includes sending search requests to the indexers for fulfillment.

[ykpramodhcbt]

We made the mistake of listing all the indexers through distsearch.conf and we started getting 3 copies of information. We later added the indexer master to SHs and it over-ruled the distsearch configuration.

We have some searches where we read from previous scheduled searches a lot. We believe that SH in our case would fetch data from indexers, get data from artifacts and does the processing.

[esix_splunk]

When you add each indexer as a search peer in distsearch.conf (or the GUI), vs joining the SH/SHC to a index cluster, you are essentially eliminating the mapreduce functionality of distributed search and the index cluster. This is because the SH/SHC no longer knows its searching a cluster, nor does it know where the searchable buckets are. So its essentially distributed to all indexers, and awaiting response. So extra compute...

Rebuild your SHC to connect to the cluster master node and let search run properly. That should alleviate load. After that, you need to check through search log and determine where your searches are slowing down. E.g., is it indexer side disk or memory, or SH. From there, adjust resources as required..