I have a saved search, which executes for every 5 minutes.
Sometimes it fails because it was running for more time(found - Running 0% in view recent of action).
If i delete that particular job and restart the saved search , its working fine.
To make it work consistency, Please suggest me a way to automatically delete long running job and restarting it.
Thanks in advance.
What does the saved search do, summary indexing, reporting? If it runs more than 5 min, first consider optimization of the query and/or reducing the frequency, e.g. every 15 or 30 mins.
If you have a query that runs every 5 minutes and that sometimes takes more than 5 minutes to complete, then that query is a strong candidate for refactoring.
Usually, when we hear this, we will find that the query is repeatedly chewing up the same data for a day, a week, a month, a quarter worth of data, and doing so every 5 minutes. Most of that is wasted effort.
If you tell us more about the underlying search, preferably posting the search code with any confidential information removed, then we can suggest whether it is a better candidate for a summary index, for search acceleration, for preprocessing into a csv file, or any of the other ways that such a search can be pre-aggregated to take literally 1/10 or less of the search time.
Okay, that does not match the usual issue.
In that code there are only three things that can take measurable time, the
| aiamurlping dedup and the two
spaths. The spaths are only going to take much time when the json is malformed, so that leaves the
So, as a test, run this query
| inputlookup web_inventory.csv | where url_type="static" AND Type_Application="Other" | stats count as dupcount by url | where dupcount>1
... and if there are any results, then add this line after the line
| table _time, category_id ..., and before the rest of your query...
| dedup url
Other than that, you are probably being killed by the actual time it takes to run aiamurlping. I can't find any information on that, although there are hints that it might be a python add-on, and by the name, it appears to be pinging a url for some kind of response.
Given this situation myself, for testing, I would be breaking that web_inventory.csv down into small chunks (say 6 chunks of 3 urls) and running them independently for a few hours to see which ones are causing the delays.
Alternatively, if you could establish that the search will ALWAYS end within x minutes, then you could clone your search and have
ceiling(x/5) identical versions that alternate, for example with every hour version 1 running at :00, :15, :30, :45; version 2 running at :05, :20, :35, :50; version 3 running at :10, :25, :40, :55)
| aiamurlping dedup - may i know what this command does.. seems like a customer command (this aiamurlping not available on search reference and google search also returns this page only)
Thanks for the comments.
|inputlookup web_inventory.csv | where url_type="static" AND Type_Application="Other" | eval content_id="static" | eval category_id="static" |eval title="static url" | eval _time=now()|table _time,category_id, content_id, title, url, id, name, description | aiamurlping dedup | eval result_code=spath(payload, "resultCode") | eval result_error_message=spath(payload, "errorDescription") | fields - payload
webinventory.csv with "Other" is having 18 URL. The query has to get the responsetime and status of the URL for every 5 min, which in turn will be index in a summary index.
In Adhoc search the query executes in 3+ sec, whereas its running for long time in saved search.
Please advise on this.
Thanks in advance
I wonder if there might be a coding or permissions issue with aiamurlping that is not showing up when run adhoc. Do you have the actual code to that command?
Why in the world would you run this every 5 minutes? Does
web_inventory.csv change more frequently than every 5 minutes? This makes no sense.
Yep, wish I had more details on that
aiamurlping... seems like he's trying to check the server response times, but I'm not sure why you'd want to have splunk itself doing that, rather than an external script. It seems like using a utility knife as a screwdriver.