topic Restart saved search when it was running for longer time in Splunk Search

Restart saved search when it was running for longer time

sujith_usha_kum — Tue, 13 Jun 2017 11:23:54 GMT

Hi All,

I have a saved search, which executes for every 5 minutes.

Sometimes it fails because it was running for more time(found - Running 0% in view recent of action).

If i delete that particular job and restart the saved search , its working fine.

To make it work consistency, Please suggest me a way to automatically delete long running job and restarting it.

Thanks in advance.

Re: Restart saved search when it was running for longer time

somesoni2 — Tue, 13 Jun 2017 16:05:54 GMT

What does the saved search do, summary indexing, reporting? If it runs more than 5 min, first consider optimization of the query and/or reducing the frequency, e.g. every 15 or 30 mins.

Re: Restart saved search when it was running for longer time

DalJeanis — Tue, 13 Jun 2017 16:10:47 GMT

If you have a query that runs every 5 minutes and that sometimes takes more than 5 minutes to complete, then that query is a strong candidate for refactoring.

Usually, when we hear this, we will find that the query is repeatedly chewing up the same data for a day, a week, a month, a quarter worth of data, and doing so every 5 minutes. Most of that is wasted effort.

If you tell us more about the underlying search, preferably posting the search code with any confidential information removed, then we can suggest whether it is a better candidate for a summary index, for search acceleration, for preprocessing into a csv file, or any of the other ways that such a search can be pre-aggregated to take literally 1/10 or less of the search time.

Re: Restart saved search when it was running for longer time

woodcock — Tue, 13 Jun 2017 17:47:10 GMT

Show us the search.

Re: Restart saved search when it was running for longer time

sujith_usha_kum — Tue, 29 Sep 2020 14:27:23 GMT

Hi all,

Thanks for the comments.

|inputlookup web_inventory.csv | where  url_type="static" AND Type_Application="Other" | eval content_id="static" | eval category_id="static" |eval title="static url" | eval _time=now()|table _time,category_id, content_id, title, url, id, name, description | aiamurlping  dedup | eval result_code=spath(payload, "resultCode")  | eval result_error_message=spath(payload, "errorDescription") | fields - payload

web_inventory.csv with "Other" is having 18 URL. The query has to get the response_time and status of the URL for every 5 min, which in turn will be index in a summary index.

In Adhoc search the query executes in 3+ sec, whereas its running for long time in saved search.

Please advise on this.

Thanks in advance

Re: Restart saved search when it was running for longer time

DalJeanis — Wed, 14 Jun 2017 15:26:39 GMT

Okay, that does not match the usual issue.

In that code there are only three things that can take measurable time, the | aiamurlping dedup and the two spaths. The spaths are only going to take much time when the json is malformed, so that leaves the aiamurlping.

So, as a test, run this query

| inputlookup web_inventory.csv 
| where  url_type="static" AND Type_Application="Other"    
| stats count as dupcount by url
| where dupcount>1

... and if there are any results, then add this line after the line | table _time, category_id ..., and before the rest of your query...

| dedup url

Other than that, you are probably being killed by the actual time it takes to run aiamurlping. I can't find any information on that, although there are hints that it might be a python add-on, and by the name, it appears to be pinging a url for some kind of response.

Given this situation myself, for testing, I would be breaking that web_inventory.csv down into small chunks (say 6 chunks of 3 urls) and running them independently for a few hours to see which ones are causing the delays.

Alternatively, if you could establish that the search will ALWAYS end within x minutes, then you could clone your search and have ceiling(x/5) identical versions that alternate, for example with every hour version 1 running at :00, :15, :30, :45; version 2 running at :05, :20, :35, :50; version 3 running at :10, :25, :40, :55)

Re: Restart saved search when it was running for longer time

DalJeanis — Wed, 14 Jun 2017 15:35:02 GMT

I wonder if there might be a coding or permissions issue with aiamurlping that is not showing up when run adhoc. Do you have the actual code to that command?

Re: Restart saved search when it was running for longer time

woodcock — Wed, 14 Jun 2017 17:25:37 GMT

Why in the world would you run this every 5 minutes? Does web_inventory.csv change more frequently than every 5 minutes? This makes no sense.

Re: Restart saved search when it was running for longer time

DalJeanis — Thu, 15 Jun 2017 02:31:35 GMT

Yep, wish I had more details on that aiamurlping... seems like he's trying to check the server response times, but I'm not sure why you'd want to have splunk itself doing that, rather than an external script. It seems like using a utility knife as a screwdriver.

Re: Restart saved search when it was running for longer time

inventsekar — Thu, 15 Jun 2017 03:32:40 GMT

| aiamurlping dedup - may i know what this command does.. seems like a customer command (this aiamurlping not available on search reference and google search also returns this page only)