Deployment Architecture

What do I do if I have too many saved searches with status="continued" in my search head cluster?

ishaanshekhar
Communicator

I have a search head cluster with quite a few saved searches that run every 5 mins.

Sometimes, the status of a few saved searches become "continued". I understand that system will come back to that later, but quite often the queue gets longer and longer and continues until I restart the SHC.

I would like to know what should I do in this scenario? options:

1) Add another SH in the cluster
2) bump resource values in limits.conf
3) increase the frequency of savedsearches to distribute the load (last option)
4) or anything else?

Thanks
Ishaan

0 Karma

teunlaan
Contributor

What you can do, depends on the reason why there are not running immediately.
So before you try to fix anything find the error.

1) look at the duration of your searches. If you run the every 5 minutes, they should be finished within the 5 minutes.
2) If searches take a long time to finish, take a look at the search. Are the build in the right way (specify host, source type, ect.)
3) Are the searches “continued” because you don’t have any CPU-cores left on you searchheads OR on you indexers.

- Maybe more Indexers will help (single indexer has to search a smaller amount of data).

- Extra Searchhead is only usefull, if your Indexers can handle it

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

The searches are actually performed on the indexers, so I would look at that first. Here is a bit of reading to get you started: http://docs.splunk.com/Documentation/Splunk/6.2.0/Deploy/Datapipeline

0 Karma
Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...