increase search performance in clustered environment

[sigma]

Hi all

I created an environment with following instances:

1. cluster master
2. three search heads
3. four indexers
4. heavy forwarder
5. license server
6. deployment server
7. deployer

We have more that 50 clients so that I deployed the deployment server on a dedicated server. We have some indexes but one of them (say index named A) have about 35K per minute events. The heavy forwarder load balances the events between four indexers. The replication factor is 4 and the search factor is 3. A simple search like 'index=A' can return about 17M events at about 5 minutes.

I want to speed up the search on the index A. I can change whole deployment and environment if anyone has an idea about speeding up the search.I would be grateful If anyone could help me about parameters like replication factor or search factor, number of indexers and... to speed up the search.

Thank you.

[bowesmana]

Echoing what @PickleRick said, you can throw hardware at solving the problem, but if your users are doing bad searches (transaction/join) or other poor subsearches, or using lots of poorly chosen wildcards or needing to use eventstats, streamstats, mvexpand, sort a lot, then you will be pushing load to the search heads from the indexers, so adding indexers will not solve the problem and adding search heads will just ensure that you will need to do it again if the same users keep writing bad searches.

You really need to look at the monitoring console to identify if there are poor searches running and look at those searches and identify what the performance issue is.

[PickleRick]

First and foremost - often the key to speeding up the search is just writing a good search.

Typically you don't just search for all events from a given index - you either look for something specific or transform the data to get some meaningful summary.

Having said that - there are so many points where things can go slow (network links, performance and resources of single hosts, data distribution, the load on your environment) that it's impossible to give a "general" answer.

So architecture is one thing (and you really should get your local friendly Splunk Partner involved to design an architecture fitting your specific needs - including resilience, HA, capacity and specific use cases) but troubleshooting existing environment is another. You can verify what your search is waiting for using the "Inspect Job" button.