What do I do if I have too many saved searches with status="continued" in my search head cluster?

[ishaanshekhar]

I have a search head cluster with quite a few saved searches that run every 5 mins.

Sometimes, the status of a few saved searches become "continued". I understand that system will come back to that later, but quite often the queue gets longer and longer and continues until I restart the SHC.

I would like to know what should I do in this scenario? options:

1) Add another SH in the cluster
2) bump resource values in limits.conf
3) increase the frequency of savedsearches to distribute the load (last option)
4) or anything else?

Thanks
Ishaan

[teunlaan]

What you can do, depends on the reason why there are not running immediately.
So before you try to fix anything find the error.

1) look at the duration of your searches. If you run the every 5 minutes, they should be finished within the 5 minutes.
2) If searches take a long time to finish, take a look at the search. Are the build in the right way (specify host, source type, ect.)
3) Are the searches “continued” because you don’t have any CPU-cores left on you searchheads OR on you indexers.

- Maybe more Indexers will help (single indexer has to search a smaller amount of data).

- Extra Searchhead is only usefull, if your Indexers can handle it

[mreynov_splunk]

The searches are actually performed on the indexers, so I would look at that first. Here is a bit of reading to get you started: http://docs.splunk.com/Documentation/Splunk/6.2.0/Deploy/Datapipeline