topic Re: How to trigger a script based on an alert and include those results in the email? in Alerting

How to trigger a script based on an alert and include those results in the email?

tkwaller — Tue, 18 Aug 2015 16:42:02 GMT

Hello

I'm trying to find a way to trigger a script based on an alert and include those results in the alert email.

Basically:
The alert looks for this error in the wso2carbon.log file:

java.io.IOException: Too many open files

This triggers an alert to send email notifiacations. What I would like to do is have this alert run

"echo there are "/usr/sbin/lsof | grep wso2am | wc -l" open wso2am files"
"echo there are "/usr/sbin/lsof | grep java | wc -l" open java files"

and include this in the email.

What would be the best way to accomplish this?
Thanks for the thoughts!

Re: How to trigger a script based on an alert and include those results in the email?

jensonthottian — Tue, 18 Aug 2015 17:12:33 GMT

Your search query should be -
*** source=“wso2carbon.log” “java.io.IOException: Too many open files” | stats count
Alert config-
Trigger Condition: Number of Results is > 0
Actions: Send Email, List in Triggered Alerts and in the alert config put a check on run a script(select ur script), link to results and inline table results in the email

Re: How to trigger a script based on an alert and include those results in the email?

tkwaller — Tue, 18 Aug 2015 19:16:58 GMT

Let me be a little more specific:

This is my search:
index=apigateway host="servername" "java.io.IOException: Too many open files".

This runs on a cron schedule of "every 10 minutes" and alerts when any results > 0 are found.
I already setup all of the alert actions.
I added the the check for "Run a script" and put the script into $SPLUNK_HOME/bin/scripts/

The alert triggers and emails as it should BUT the results of the script are not included in the email. Only the results of the search.

Re: How to trigger a script based on an alert and include those results in the email?

jensonthottian — Tue, 18 Aug 2015 19:29:31 GMT

I dont think you can have the results of your script as part of the original email alert. What you can do is to have an additional line of code in your script to have the output emailed to the DL you want but let me remind you that this will be a separate email altogether.

Re: How to trigger a script based on an alert and include those results in the email?

tkwaller — Tue, 18 Aug 2015 20:47:12 GMT

Thats what I was thinking but wasn't sure if there was another way.