topic How to create an alert to include client names, error counts, and the error log events that triggered the alert? in Alerting

How to create an alert to include client names, error counts, and the error log events that triggered the alert?

AmitKrJash — Tue, 29 Dec 2015 10:26:32 GMT

Hi,

I have created an alert where it checks the status of the client accessing the application. The status will be either ERROR or SUCCESS. If the status is SUCCESS, then it is fine, but if it is ERROR, it should count the number of ERROR for particular clients and will pop up an email as an alert with the list of clients and their respective ERROR counts.

This is the search:

index="abc" sourcetype="xyz" STATUS=ERROR CLIENT_ID=*|stats count by CLIENT_ID

For example lets say I have 2 clients: Amit@xyz.com and Jash@xyz.com
So the final output which I am receiving as an alert for every 1 hour are:

Client_ID Count of ERROR
Amit@xyz.com 2
Jash@xyz.com 1

So that means there are 2 errors for Amit@xyz.com and 1 error for Jash@xyz.com in the log files. So lets say the errors in the log files are:

    2015-12-29 04:05:25 ERROR - [Client ID: Amit@xyz.com] - Client is not configured properly in the database
    2015-12-29 04:06:32 ERROR - [Client ID: Amit@xyz.com] - Client is not having enough permission to access the application

2015-12-29 04:07:21 ERROR - [Client ID: Jash@xyz.com] - SOAP Fault occurred

My question is, isthere any way to email these above mentioned log file lines along with the ERROR counts for clients in the alert? So the final alert which I should receive should be something like this:

Client_ID Count of ERROR
Amit@xyz.com 2
Jash@xyz.com 1

2015-12-29 04:05:25 ERROR - [Client ID: Amit@xyz.com] - Client is not configured properly in the database
2015-12-29 04:06:32 ERROR - [Client ID: Amit@xyz.com] - Client is not having enough permission to access the application

2015-12-29 04:07:21 ERROR - [Client ID: Jash@xyz.com] - SOAP Fault occurred

Any help on this will be greatly appreciated.

Re: How to create an alert to include client names, error counts, and the error log events that triggered the alert?

jplumsdaine22 — Tue, 29 Dec 2015 13:26:09 GMT

If you're happy for them to be in line with with counts Just add the raw values into stats:

index="abc" sourcetype="xyz" STATUS=ERROR CLIENT_ID=*|stats count values(_raw)  by CLIENT_ID

See the description for values(X) in http://docs.splunk.com/Documentation/Splunk/6.3.2/SearchReference/CommonStatsFunctions

Re: How to create an alert to include client names, error counts, and the error log events that triggered the alert?

AmitKrJash — Mon, 11 Jan 2016 16:04:34 GMT

@jplumsdaine22 Thanks a lot for the help.. It is working for me now. I am able to pull up the entire Error logs in the alert. Is there a way to get the source and the host details in the alert mail from where the logs are getting generated. In the alert mail body I tried something like this

Source Log= '$source$'
Server = '$host$'
but it is not working.