Alerting

How can I set up a script to run after an alert is triggered?

DAnkita
New Member

I want to run a script after a particular alert triggers, taking server names as input from the Splunk alert result.
The alerts result is in bellow format:
time server LoadFailed date

I want to take server list from here and execute a command on all the servers listed .

I have gone through http://docs.splunk.com/Documentation/Splunk/5.0/Alert/Configuringscriptedalerts document , but couldn't implement much .
Just could get a idea that may we we need to open file result and grep the argument .
Please help with the script.

Thanks
AD

0 Karma

nickhills
Ultra Champion

A really quick and dirty method is to do this in Bash - if nothing else it will help you get to grips with how the process works.

my_custom_action.sh

#!/bin/bash
ResultsList=$(cat $8|gzip -d|tail -n +2)
echo $ResultsList > output.txt

Taking each command as it comes:

ResultsList will contain the results of your Splunk search
cat $8 when splunk executes your script the $8 parameter will be the path of the search results on your server. Cat will output the contents of the file and pass it to:
gzip -d - this will decompress the results to make them readable, and then:
tail -n +2 - will ignore the top line which is the header, and start reading from line 2!
echo $ResultsList will write the results of the above into output.txt for you to review.

If my comment helps, please give it a thumbs up!
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @DAnkita,

Which version of splunk version are you running ? Because Scripted Alerts are deprecated since Splunk 6.3 and this feature is replaced with Custom Alert Action so I'll recommend you to create custom alert action based on your requirement if you are running Splunk Version 6.3 or higher. In given Custom Alert Action link, splunk also provided 2-3 example so you refer those as well.

I hope this helps.

Thanks,
Harshil

0 Karma

DAnkita
New Member

Hi Harshil ,

Thanks for your reply , I m using Splunkweb7 currently .

0 Karma

DAnkita
New Member

My main problem is how can we take the server list as a input , if you can help .

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

In Custom Alert action when you fetch payload, you will able to see results_file which generates when your schedule search will run and it will store output value in csv format but file will be in compressed .gz format.

So high level steps in your script for Custom Alert Action
1.) Fetch results_file from payload
2.) zcat results_file and find count of rows without header and store it in variable
3.) Run for loop with count which you found in Step 2 and read rows one by one and fetch server from that row value and do necessary action based on your requirement.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...