Getting Data In

Complex Search causes Script Alert Action to not fire

richnavis
Contributor

Hi All,

I ran into an issue where certain searches seem to caused scripted alert actions to fail. In trying to figure out what was wrong, I created a VERY basic search, and a VERY basic scripted alert... Essentially, echo.bat, which echos some parameters to a local splunk drive. I configured the alert to ALWAYS fire, regardless of the resuls, and also configured an email alert so I'd get double verification that the search was running. With the simple search, this works exactly as expected.. So I simply CLONED the search, and then replaced the simple search with my more complex search.. I still get the email every minute, but the echo.bat does not work. Seems to me that their is a bug in the system.. anyone else run across this? BTW.. here's the searches I used...

Search 1

exception | stats count by host

Search 2

index=summary search_name=IIS* earliest=-2d@d latest=-1d@d | regex search_name="IIS_(ORDER|PRODUCT|WWW)" | stats avg(response_time) as rt_yesterday by search_name
|append maxtime=600 [search index=summary search_name=IIS* earliest=-60m@m latest=-0m@m | regex search_name="IIS_(ORDER|PRODUCT|WWW)" | stats avg(response_time) as rt_today by search_name] | stats values(rt_yesterday) as yesterday values(rt_today) as today by search_name | eval %change = (today-yesterday)/yesterday*10

Tags (2)
0 Karma

richnavis
Contributor

BUG! Turns out that by eliminating the regex part of the search, I can get this to work. Turns out that replacing regex search_name="(PRVD|ORDER|PRODUCT) with three explicit references to the search_name seperated by OR will get the script to fire. This is definitely a bug in their code (python perhaps) that their dev team should be aware of..

0 Karma

richnavis
Contributor

as mentioned, the file gets generated if I simplify the search, therefore not a permissions issue. I have checked permissions (several times)

0 Karma

linu1988
Champion

Did you check the permission on the folder?

"%SPLUNK_HOME%\bin\scripts\echo_output.txt"

Replace with some other location and file outside Splunk directory. I have seen this happens due to lack of permission on the directory.

0 Karma

richnavis
Contributor

yes... echo.bat doesn't generate the output file in the folder. If I modify replace search 2 with search 1, then the output file gets generated. The search is setup to ALWAYS alert, so it doesn't matter whether the search generates results or not... However, I've also checked many times and search 2 consistently returns results when run from the search bar. I suspect that this is a bug, and splunk support is working on it, but not much progress in weeks.. 😞

0 Karma

linu1988
Champion

Are you suggesting the echo.bat doesn't generate your output file in the folder? Did you check if splunk has permission on the folder where the bat file is present?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...