Do you use the Splunk on Splunk (SoS) app? It can help provide insights into both your search heads and indexer performance. It's new to me but the search activity app can help identify some badly performing searches. And of course there's the *nix and Windows apps to look at OS performance issues.
I also like reviewing some of the past conference papers at places like
including the Jiffy Lube Quick Tune-Up and Splunk Monitoring Console presentations.
It could be dependent on many things, your indexing volume per day, the number of indexers, their hardware and performance, the number of cores in your search head, as well as the type of search being run. Could you specify the amount of indexing volume (or your license volume per day) as well as the number of indexers, the hardware assigned to your search head and an example of a slow search?
If possible, you should provide more information.
Do you use a single search head? Are your searches slow? Is the response time of the web ui slow? have you taken a look at the performance of your machine? Is there a high cpu / memory utilization? Is it one certain search that is slow? etc...
Cpu and memory utilization are below 40%. yes UI part of search head is very slow.because the macro and saved search which we use in run time are very big. that could be the reason i guess.
Is there any way to optimize this response time?
So it is fast to open a dashboad or go to the settings etc ?
But you have to wait a long time until a certain big search is returning results?
If so, it depends on your search. Maybe we can help you optimize it. Therefore, you have to provide more information about this specific search if possible.