Time to ask the experts out there;
Is it possible to create a scheduled search, and based on some criteria create an email alert, but before sending the alert out, run a custom Python script, collect specific data from another system, add that data to the alert email and send all that out? I've been going through the documentation and can't find the the specific information I'm looking for. I'm looking for a simple yes/no answer and then if yes, where to find the documentation.
Thanks in advance
Email and script are two different parallel actions for an alert. You can not make them interdependent. Perhaps you should use the script itself to send an email instead of splunk email.
If I understand it correctly then, once an alert is triggered from a search I can send that information in an email notification or launch a script. Not both. Damn.
Is there any way that I can trigger a script to run in the search then, and add the resulting data to the alert?
You can run both, but they will happen at the same time. You may want to try two alerts. One that runs the script and possibly creates a new event in Splunk. A second alert could trigger off of the new event which would have the info you need. Probably other ways to do it, but this could be an option for you.
So the process steps would be;
1. The search would run and trigger an alert
2. That alert would trigger the script passing data to it - no email
3. The script would run, collect additional data and trigger a second alert. This second alert would then finally send the email notice.
Whoa, and Splunk is not set up to run a script within the search first to avoid all the other steps - right?
OK, I think I found something called "External Fields Lookup" in the documentation. It describes the Python script called "external_lookup.py" that is shipped with Splunk already configured. I have used this previously and it works. Basically one passes an IP and the script will send back a hostname. The call is "sourcetype=access_combined | eval clienthost = host | lookup dnslookup clienthost | stats count by clientip". I can use that as a model and instead of making a DNS call, I'll run my internal call in the Python script. This way I can collect the required data within the search prior to creating the alert.