Alerting

How to : Schedule an Alert everytime a job fails more than once within an hour

christinaef07
Loves-to-Learn Everything

Hello, I am trying to create an Alert on Splunk. I want to create an alert so that I am alerted every time a job fails 2 times or more within an hour. We have several different jobs running. Right now, I have a table displaying each job with the amount of failures of each. 

 

 

index=?? uuid=* |search status=success | rex "message=(?<message>.*)" | stats count(eval(status=="failed")) AS Failures by workflow_name | table workflow_name, Failures

 

 

This displays something like : 

workflow_name        Failures

workflow_1                 3

workflow_2                 1

workflow_3                7

How can I fix this to filter and only include the workflows that have failed more than once (workflow_1 & workflow_3) and within a specific time frame - 1 hr.  Additionally, I want to pull in info about the specific workflow with the latest failure (for ex: message, uuid, etc). For ex:

 

workflow_name        Failures.       Latest message       Latest uuid 

workflow_1                 3                        error msg                    12345678

workflow_3                7                          error msg                  98765432

 

Labels (3)
0 Karma

aohls
Contributor

A where clause at the end of you query should do it; | where Failures > 1. Then you could schedule the job to run on whatever time frame you need.

0 Karma
Get Updates on the Splunk Community!

Demo Day: Strengthen Your SOC with Splunk Enterprise Security 8.1

Today’s threat landscape is more complex than ever. Security operation centers (SOCs) are overwhelmed with ...

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...