I ran into an issue where certain searches seem to caused scripted alert actions to fail. In trying to figure out what was wrong, I created a VERY basic search, and a VERY basic scripted alert... Essentially, echo.bat, which echos some parameters to a local splunk drive. I configured the alert to ALWAYS fire, regardless of the resuls, and also configured an email alert so I'd get double verification that the search was running. With the simple search, this works exactly as expected.. So I simply CLONED the search, and then replaced the simple search with my more complex search.. I still get the email every minute, but the echo.bat does not work. Seems to me that their is a bug in the system.. anyone else run across this? BTW.. here's the searches I used...
exception | stats count by host
index=summary search_name=IIS* earliest=-2d@d latest=-1d@d | regex search_name="IIS_(ORDER|PRODUCT|WWW)" | stats avg(response_time) as rt_yesterday by search_name
|append maxtime=600 [search index=summary search_name=IIS* earliest=-60m@m latest=-0m@m | regex search_name="IIS_(ORDER|PRODUCT|WWW)" | stats avg(response_time) as rt_today by search_name] | stats values(rt_yesterday) as yesterday values(rt_today) as today by search_name | eval %change = (today-yesterday)/yesterday*10
BUG! Turns out that by eliminating the regex part of the search, I can get this to work. Turns out that replacing regex search_name="(PRVD|ORDER|PRODUCT) with three explicit references to the search_name seperated by OR will get the script to fire. This is definitely a bug in their code (python perhaps) that their dev team should be aware of..
Did you check the permission on the folder?
Replace with some other location and file outside Splunk directory. I have seen this happens due to lack of permission on the directory.
yes... echo.bat doesn't generate the output file in the folder. If I modify replace search 2 with search 1, then the output file gets generated. The search is setup to ALWAYS alert, so it doesn't matter whether the search generates results or not... However, I've also checked many times and search 2 consistently returns results when run from the search bar. I suspect that this is a bug, and splunk support is working on it, but not much progress in weeks.. 😞