Alerting

Create alert when same warning keep coming for more than 5 hours

JYTTEJ
Communicator

During maintenance we get the same application alert: MON001E on each trx. I want to set up an alert if application alert MON001E keep coming for more than 5 hours.

The search is to be scheduled to run each hour 24/7

If this application alert (MON001E) keep coming for more than 5 hours, SPLUNK must give an alert.

Example:no SPLUNK alert:

First MON001E at 01:00:00

Last MON001E at 05:00:00

Example: SPLUNK Alert:

First MON001E at 02:00

Last MON001E at 09:00

SPLUNK must alert at 08:00

Will this search do the trick?:

MID=MON001E earliest-6h latest=-1h

Tags (1)
0 Karma
1 Solution

joelshprentz
Path Finder

Try this search:

MID=MON001E earliest=-6h@h latest=@h | timechart span=1h count | eval count=min(1, count) | stats sum(count) as alertHours

Send an alert whenever alertHours > 5.

The search collects six hours of data matching MON001E. The timechart counts how many matches occur in each of the six hours. The eval limits those counts to be either 0 (no matches) or 1 (at least one MON001E). The stats totals the number of hours with at least one MON001E.

I interpreted your question as, "Send an alert whenever MID=MON001E occurs in six consecutive hours."

View solution in original post

joelshprentz
Path Finder

Try this search:

MID=MON001E earliest=-6h@h latest=@h | timechart span=1h count | eval count=min(1, count) | stats sum(count) as alertHours

Send an alert whenever alertHours > 5.

The search collects six hours of data matching MON001E. The timechart counts how many matches occur in each of the six hours. The eval limits those counts to be either 0 (no matches) or 1 (at least one MON001E). The stats totals the number of hours with at least one MON001E.

I interpreted your question as, "Send an alert whenever MID=MON001E occurs in six consecutive hours."

JYTTEJ
Communicator

great! Thank you very much

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...