Alerting

Splunk and Custom Scripts

OldManEd
Builder

Time to ask the experts out there;

Is it possible to create a scheduled search, and based on some criteria create an email alert, but before sending the alert out, run a custom Python script, collect specific data from another system, add that data to the alert email and send all that out? I've been going through the documentation and can't find the the specific information I'm looking for. I'm looking for a simple yes/no answer and then if yes, where to find the documentation.

Thanks in advance

0 Karma
1 Solution

OldManEd
Builder

Creating a dynamic lookup with a python script solved this problem.

View solution in original post

0 Karma

OldManEd
Builder

Creating a dynamic lookup with a python script solved this problem.

0 Karma

pradeepkumarg
Influencer

Email and script are two different parallel actions for an alert. You can not make them interdependent. Perhaps you should use the script itself to send an email instead of splunk email.

0 Karma

OldManEd
Builder

If I understand it correctly then, once an alert is triggered from a search I can send that information in an email notification or launch a script. Not both. Damn.

Is there any way that I can trigger a script to run in the search then, and add the resulting data to the alert?

0 Karma

kmorris_splunk
Splunk Employee
Splunk Employee

You can run both, but they will happen at the same time. You may want to try two alerts. One that runs the script and possibly creates a new event in Splunk. A second alert could trigger off of the new event which would have the info you need. Probably other ways to do it, but this could be an option for you.

0 Karma

OldManEd
Builder

So the process steps would be;
1. The search would run and trigger an alert
2. That alert would trigger the script passing data to it - no email
3. The script would run, collect additional data and trigger a second alert. This second alert would then finally send the email notice.

Whoa, and Splunk is not set up to run a script within the search first to avoid all the other steps - right?

0 Karma

OldManEd
Builder

OK, I think I found something called "External Fields Lookup" in the documentation. It describes the Python script called "external_lookup.py" that is shipped with Splunk already configured. I have used this previously and it works. Basically one passes an IP and the script will send back a hostname. The call is "sourcetype=access_combined | eval clienthost = host | lookup dnslookup clienthost | stats count by clientip". I can use that as a model and instead of making a DNS call, I'll run my internal call in the Python script. This way I can collect the required data within the search prior to creating the alert.

OldManEd
Builder

Yup, this worked.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...