Feedback
Got feedback? We want it! Submit your comments and suggestions for our community here.

Need to Setup splunk alert for total percentage jobs failure.

alamma8
Loves-to-Learn

Using EMR Spark & all the logs goes to splunk & there are multiple type of jobs running in the cluster. I want to setup splunk alert,if more that 5% total no. of jobs failed then we get the alert.

0 Karma

alamma8
Loves-to-Learn

Hi,

Thanks for you reply.

using following query i am able to get all the failed jobs.

index=emr | search applicationType=SPARK finalStatus=FAILED

In our environment more than 300 jobs are running per day.(batch jobs & streaming jobs).

#1:-I want to setup an alert, if fail jobs count reach to 5% then it trigger the alert.

#2:-Numbers of job can be fluctuate. some day total count would be more than 300 or less than 300. So percent (5%) should be on actual count. like total count of the day is 280. then what would be the parameter.

Please give me the query what should i need to run.

Thanks in Advanced

0 Karma

alamma8
Loves-to-Learn

Hi,

Thanks for you reply.

using following query i am able to get all the failed jobs.

index=emr | search applicationType=SPARK finalStatus=FAILED

In our environment more than 300 jobs are running per day.(batch jobs & streaming jobs).

#1:-I want to setup an alert, if fail jobs count reach to 5% then it trigger the alert.

#2:-Numbers of job can be fluctuate. some day total count would be more than 300 or less than 300. So percent (5%) should be on actual count. like total count of the day is 280. then what would be the parameter.

Please give me the query what should i need to run.

Thanks in Advanced.

 

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

In general, since the question does not contain specifics about the data, you'll need a count of all jobs as well as a count of jobs that failed.  Use math to find the failure percentage.

your search | eval fail = if(test for failure, 1, 0)
| stats count as total, sum(fail) as failures
| eval pct = failures * 100 / total
| where pct > 5
---
If this reply helps you, an upvote would be appreciated.
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.