topic Re: Set timeout for saved search run in Getting Data In

Set timeout for saved search run

kkos94 — Tue, 28 May 2019 09:51:54 GMT

Hello splunk community!

Is there any way to add a timeout to a saved search so that it can fail if it runs for too long?

In case this is not possible, is there another way for me to get notified when a search has been running for longer than it should(let's say, 1 hour).

Any ideas would be appreciated, thanks!

Re: Set timeout for saved search run

koshyk — Tue, 28 May 2019 10:17:10 GMT

You have few options available in savedsearches.conf like dispatch.max_time , auto_summarize.max_time as i'm not sure where the delay happens?

Alerting long runing queries/savedsearches are pretty straight forward. If you have monitoring Console, then the searches are already built in en-US/app/splunk_monitoring_console/search_usage_statistics_deployment., especially "Long-running Searches". You can configure alerting for any of those

Essentially the base query would look like..

(index=_audit search_group=dmc_group_search_head search_group=* action=search sourcetype=audittrail search_id!="rsa_*") 
| eval search_type=case(match(search_id,"^SummaryDirector_"),"summarization",match(search_id,"^((rt_)?scheduler__|alertsmanager_)"),"scheduled",match(search_id,"\\d{10}\\.\\d+(_[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})?$"),"ad hoc",true(),"other") 
| eval search=if((isnull(savedsearch_name) OR (savedsearch_name == "")),search,savedsearch_name) 
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host
| where total_run_time>3600

Re: Set timeout for saved search run

DavidHourani — Tue, 28 May 2019 10:23:09 GMT

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

Re: Set timeout for saved search run

kkos94 — Tue, 28 May 2019 10:58:22 GMT

dispatch.max_time did exactly what I needed to make it work.

Thanks a lot!

Re: Set timeout for saved search run

DavidHourani — Tue, 28 May 2019 11:00:18 GMT

You're welcome !

Re: Set timeout for saved search run

kkos94 — Tue, 28 May 2019 11:43:21 GMT

Thanks for your reply!

Turns out I could modify dispatch.max_time for a specific saved search instead of modifying it in the .conf file.

Good point on configuring an alert though. I will most definitely need it in the future.