Alerting

How to trigger a script based on an alert and include those results in the email?

Builder

Hello

I'm trying to find a way to trigger a script based on an alert and include those results in the alert email.

Basically:
The alert looks for this error in the wso2carbon.log file:

java.io.IOException: Too many open files 

This triggers an alert to send email notifiacations. What I would like to do is have this alert run

"echo there are "/usr/sbin/lsof | grep wso2am | wc -l" open wso2am files"
"echo there are "/usr/sbin/lsof | grep java | wc -l" open java files"

and include this in the email.

What would be the best way to accomplish this?
Thanks for the thoughts!

0 Karma
1 Solution

Contributor

Your search query should be -
*** source=“wso2carbon.log” “java.io.IOException: Too many open files” | stats count
Alert config-
Trigger Condition: Number of Results is > 0
Actions: Send Email, List in Triggered Alerts and in the alert config put a check on run a script(select ur script), link to results and inline table results in the email

View solution in original post

Contributor

Your search query should be -
*** source=“wso2carbon.log” “java.io.IOException: Too many open files” | stats count
Alert config-
Trigger Condition: Number of Results is > 0
Actions: Send Email, List in Triggered Alerts and in the alert config put a check on run a script(select ur script), link to results and inline table results in the email

View solution in original post

Builder

Let me be a little more specific:

This is my search:
index=apigateway host="servername" "java.io.IOException: Too many open files".

This runs on a cron schedule of "every 10 minutes" and alerts when any results > 0 are found.
I already setup all of the alert actions.
I added the the check for "Run a script" and put the script into $SPLUNK_HOME/bin/scripts/

The alert triggers and emails as it should BUT the results of the script are not included in the email. Only the results of the search.

0 Karma

Contributor

I dont think you can have the results of your script as part of the original email alert. What you can do is to have an additional line of code in your script to have the output emailed to the DL you want but let me remind you that this will be a separate email altogether.

Builder

Thats what I was thinking but wasn't sure if there was another way.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!