Getting Data In

Set timeout for saved search run

kkos94
Explorer

Hello splunk community!

Is there any way to add a timeout to a saved search so that it can fail if it runs for too long?

In case this is not possible, is there another way for me to get notified when a search has been running for longer than it should(let's say, 1 hour).

Any ideas would be appreciated, thanks!

Tags (2)
0 Karma
1 Solution

DavidHourani
Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

View solution in original post

kkos94
Explorer

dispatch.max_time did exactly what I needed to make it work.

Thanks a lot!

0 Karma

DavidHourani
Super Champion

You're welcome !

koshyk
Super Champion

You have few options available in savedsearches.conf like dispatch.max_time , auto_summarize.max_time as i'm not sure where the delay happens?

Alerting long runing queries/savedsearches are pretty straight forward. If you have monitoring Console, then the searches are already built in en-US/app/splunk_monitoring_console/search_usage_statistics_deployment., especially "Long-running Searches". You can configure alerting for any of those

Essentially the base query would look like..

(index=_audit search_group=dmc_group_search_head search_group=* action=search sourcetype=audittrail search_id!="rsa_*") 
| eval search_type=case(match(search_id,"^SummaryDirector_"),"summarization",match(search_id,"^((rt_)?scheduler__|alertsmanager_)"),"scheduled",match(search_id,"\\d{10}\\.\\d+(_[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})?$"),"ad hoc",true(),"other") 
| eval search=if((isnull(savedsearch_name) OR (savedsearch_name == "")),search,savedsearch_name) 
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host
| where total_run_time>3600

kkos94
Explorer

Thanks for your reply!

Turns out I could modify dispatch.max_time for a specific saved search instead of modifying it in the .conf file.

Good point on configuring an alert though. I will most definitely need it in the future.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!