Getting Data In

Set timeout for saved search run

Explorer

Hello splunk community!

Is there any way to add a timeout to a saved search so that it can fail if it runs for too long?

In case this is not possible, is there another way for me to get notified when a search has been running for longer than it should(let's say, 1 hour).

Any ideas would be appreciated, thanks!

Tags (2)
0 Karma
1 Solution

Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

View solution in original post

Super Champion

Hi @kkos94,

Definitely, you can limit the max time for a savedsearch, so you're looking for dispatch.max_time:

dispatch.max_time = <integer>
* Indicates the maximum amount of time (in seconds) before finalizing the
  search.
* Defaults to 0.

Official documentation here :
https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/Savedsearchesconf

You can set that up in either in savedsearch.conf file or by going into the advanced setting of your report from the GUI.

To get a list of all your long running searches you can use a simple search like this :

 |rest /services/search/jobs splunk_server=local

More info here :
https://answers.splunk.com/answers/508420/identify-searches-that-take-long-time-in-a-sh-clus.html
Or you could go to your monitoring console if that is configured, a lot of great info about what's happening on your search heads there.

Let me know if you need more help!

Cheers,
David

View solution in original post

Explorer

dispatch.max_time did exactly what I needed to make it work.

Thanks a lot!

0 Karma

Super Champion

You're welcome !

Super Champion

You have few options available in savedsearches.conf like dispatch.max_time , auto_summarize.max_time as i'm not sure where the delay happens?

Alerting long runing queries/savedsearches are pretty straight forward. If you have monitoring Console, then the searches are already built in en-US/app/splunk_monitoring_console/search_usage_statistics_deployment., especially "Long-running Searches". You can configure alerting for any of those

Essentially the base query would look like..

(index=_audit search_group=dmc_group_search_head search_group=* action=search sourcetype=audittrail search_id!="rsa_*") 
| eval search_type=case(match(search_id,"^SummaryDirector_"),"summarization",match(search_id,"^((rt_)?scheduler__|alertsmanager_)"),"scheduled",match(search_id,"\\d{10}\\.\\d+(_[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12})?$"),"ad hoc",true(),"other") 
| eval search=if((isnull(savedsearch_name) OR (savedsearch_name == "")),search,savedsearch_name) 
| stats min(_time) as _time, values(user) as user, max(total_run_time) as total_run_time, first(search) as search, first(search_type) as search_type, first(apiStartTime) as apiStartTime, first(apiEndTime) as apiEndTime by search_id, host
| where total_run_time>3600

Explorer

Thanks for your reply!

Turns out I could modify dispatch.max_time for a specific saved search instead of modifying it in the .conf file.

Good point on configuring an alert though. I will most definitely need it in the future.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!