Alerting

How to send alert for consecutive field values failure

manish_singh_77
Builder

Hi All,

I have a table where we have job_result field, whenever job_result field has "failure" value for consecutive 5 times then we need to set up an alert.

Example

job job _name time job result
abc abc_job 14:20 Failure
def def_job 15:30 Failure
ghi ghi_job 12:22 Failure
annc annc_job 14:20 Failure
defff defff_job 15:30 Failure
ghhhi ghhhi_job 12:22 Failure

Tags (2)
0 Karma

to4kawa
Ultra Champion
| makeresults 
| eval _raw="job   job_name  time job_result
abc   abc_job   14:20 Failure
def   def_job   15:30 Failure
ghi   ghi_job   12:22 Failure
annc  annc_job  14:20 Failure
defff defff_job 15:30 Failure
ghhhi ghhhi_job 12:22 Failure
abc   abc_job   14:20 Success
def   def_job   15:30 Failure
ghi   ghi_job   12:22 Failure
annc  annc_job  14:20 Failure
defff defff_job 15:30 Failure
ghhhi ghhhi_job 12:22 Failure"
| multikv forceheader=1 
| table job job_name time job_result
| streamstats reset_on_change=t count(eval(job_result="Failure")) as fail_count by job_result
| eventstats max(fail_count) as alert_count

Hi, @manish_singh_777
this is sample code.

your_search
| table job job_name time job_result
| streamstats reset_on_change=t count(eval(job_result="Failure")) as fail_count by job_result
| eventstats max(fail_count) as alert_count

alert_count > 5, fire alert.

manish_singh_77
Builder

Hi @to4kawa

I want to send an alert if specific job is failing for 5 consecutive times.

0 Karma

to4kawa
Ultra Champion
your_search
 | table job job_name time job_result
 | streamstats reset_on_change=t count(eval(job="abc" AND job_result="Failure")) as fail_count by job_result
 | eventstats max(fail_count) as alert_count

please add job.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...