topic Re: Splunk and Custom Scripts in Alerting

Splunk and Custom Scripts

OldManEd — Wed, 11 Jul 2018 13:50:08 GMT

Time to ask the experts out there;

Is it possible to create a scheduled search, and based on some criteria create an email alert, but before sending the alert out, run a custom Python script, collect specific data from another system, add that data to the alert email and send all that out? I've been going through the documentation and can't find the the specific information I'm looking for. I'm looking for a simple yes/no answer and then if yes, where to find the documentation.

Thanks in advance

Re: Splunk and Custom Scripts

pradeepkumarg — Wed, 11 Jul 2018 13:59:39 GMT

Email and script are two different parallel actions for an alert. You can not make them interdependent. Perhaps you should use the script itself to send an email instead of splunk email.

Re: Splunk and Custom Scripts

OldManEd — Wed, 11 Jul 2018 14:19:12 GMT

If I understand it correctly then, once an alert is triggered from a search I can send that information in an email notification or launch a script. Not both. Damn.

Is there any way that I can trigger a script to run in the search then, and add the resulting data to the alert?

Re: Splunk and Custom Scripts

kmorris_splunk — Wed, 11 Jul 2018 14:29:50 GMT

You can run both, but they will happen at the same time. You may want to try two alerts. One that runs the script and possibly creates a new event in Splunk. A second alert could trigger off of the new event which would have the info you need. Probably other ways to do it, but this could be an option for you.

Re: Splunk and Custom Scripts

OldManEd — Wed, 11 Jul 2018 14:52:23 GMT

So the process steps would be;
1. The search would run and trigger an alert
2. That alert would trigger the script passing data to it - no email
3. The script would run, collect additional data and trigger a second alert. This second alert would then finally send the email notice.

Whoa, and Splunk is not set up to run a script within the search first to avoid all the other steps - right?

Re: Splunk and Custom Scripts

OldManEd — Tue, 29 Sep 2020 20:24:49 GMT

OK, I think I found something called "External Fields Lookup" in the documentation. It describes the Python script called "external_lookup.py" that is shipped with Splunk already configured. I have used this previously and it works. Basically one passes an IP and the script will send back a hostname. The call is "sourcetype=access_combined | eval clienthost = host | lookup dnslookup clienthost | stats count by clientip". I can use that as a model and instead of making a DNS call, I'll run my internal call in the Python script. This way I can collect the required data within the search prior to creating the alert.

Re: Splunk and Custom Scripts

OldManEd — Wed, 25 Jul 2018 20:34:30 GMT

Yup, this worked.

Re: Splunk and Custom Scripts

OldManEd — Wed, 25 Jul 2018 20:39:06 GMT

Creating a dynamic lookup with a python script solved this problem.