Alerting

Create alert when same warning keep coming for more than 5 hours

JYTTEJ
Communicator

During maintenance we get the same application alert: MON001E on each trx. I want to set up an alert if application alert MON001E keep coming for more than 5 hours.

The search is to be scheduled to run each hour 24/7

If this application alert (MON001E) keep coming for more than 5 hours, SPLUNK must give an alert.

Example:no SPLUNK alert:

First MON001E at 01:00:00

Last MON001E at 05:00:00

Example: SPLUNK Alert:

First MON001E at 02:00

Last MON001E at 09:00

SPLUNK must alert at 08:00

Will this search do the trick?:

MID=MON001E earliest-6h latest=-1h

Tags (1)
0 Karma
1 Solution

joelshprentz
Path Finder

Try this search:

MID=MON001E earliest=-6h@h latest=@h | timechart span=1h count | eval count=min(1, count) | stats sum(count) as alertHours

Send an alert whenever alertHours > 5.

The search collects six hours of data matching MON001E. The timechart counts how many matches occur in each of the six hours. The eval limits those counts to be either 0 (no matches) or 1 (at least one MON001E). The stats totals the number of hours with at least one MON001E.

I interpreted your question as, "Send an alert whenever MID=MON001E occurs in six consecutive hours."

View solution in original post

joelshprentz
Path Finder

Try this search:

MID=MON001E earliest=-6h@h latest=@h | timechart span=1h count | eval count=min(1, count) | stats sum(count) as alertHours

Send an alert whenever alertHours > 5.

The search collects six hours of data matching MON001E. The timechart counts how many matches occur in each of the six hours. The eval limits those counts to be either 0 (no matches) or 1 (at least one MON001E). The stats totals the number of hours with at least one MON001E.

I interpreted your question as, "Send an alert whenever MID=MON001E occurs in six consecutive hours."

JYTTEJ
Communicator

great! Thank you very much

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...