topic Re: How to trigger an alert when status field is true for more than 5 min no matter the amount of event in Splunk Search

How to trigger an alert when status field is true for more than 5 min no matter the amount of event

Cheng2Ready — Thu, 15 Aug 2024 21:48:25 GMT

I have search query, if the Status is field is true for more than 5 min, I need to trigger an alert no matter the Event count result. if its within the timeframe then fire.
Mabey even have it search for every 1minute.

for example this should not fire an Alert because it recovered within the 5 min

1:00 Status = Down (event result count X5)
1:03 Status = up
1:07 Status = Down (event count X3)
1:10 Status = up
1:13 Status = up
1:16 Status = up

for example this should fire an Alert

1:00 Status = Down (event result count X1)
1:03 Status = Down (event result count X1)
1:07 Status = Down (event result count X1)
1:10 Status = up
1:13 Status = up
1:16 Status = up

Re: How to trigger an alert when status field is true for more than 5 min no matter the amount of event

ITWhisperer — Thu, 15 Aug 2024 23:06:43 GMT

Your data does not match your description - the Status field appears to be either "up" or "Down" not "true" - because of this, it is not clear whether you want an alert if there has been a period of at least 5 minutes of Status being "Down" or Status being "up" anywhere within the time period of the search - please clarify your requirement

Re: How to trigger an alert when status field is true for more than 5 min no matter the amount of event

Cheng2Ready — Thu, 15 Aug 2024 23:09:32 GMT

@ITWhisperer want an alert if there has been a period for every1 minute of at least 5 minutes of Status being "Down" and if its interrupted with a status = Up then it resets the count and will not alert regarding the amount of event counts

Re: How to trigger an alert when status field is true for more than 5 min no matter the amount of event

Cheng2Ready — Thu, 15 Aug 2024 23:23:28 GMT

@ITWhisperer This is what I imagine it should look like
but im not sure if there is a way to add in a condition for Streamstats
for this command? or a workaround?
"reset_on_change= if (status="UP", 1, 0) "

| bucket span=1m _time
| eval status_change=if(status="DOWN",1,0)
| streamstats sum(status_change) as down_count reset_on_change= if (status="UP", 1, 0)
| eval is_alert=if(down_count >=5 AND status="DOWN",1,0)
| where is_alert=1