How to create an alert triggered by the presence of a specific syslog message, but absence of another within a five minute interval?

Splunk Employee
Splunk Employee

Looking for assistance in crafting a scheduled search that sends a notification when I see a specific syslog message AND the absence of a specific second message in a given time period (say 5 minutes).


2014-05-10 11:30:00 NOTICE System restarted bla bla bla bla (This gets logged)
2014-05-10 11:30:15 INFO This is another log
2014-05-10 11:30:40 INFO This is a another log
2014-05-10 11:31:00 INFO All good dude ( I want to get notified when I don't get this)

0 Karma


You could do something like this:

base search with "Message One" OR "Message Two" | stats count(eval(match(_raw, "Message One"))) as ones count(eval(match(_raw, "Message Two"))) as twos

That'll give you two counts that you can use in your alert trigger condition, something like this:

where ones > 0 AND twos = 0

That'll trigger if message one is present but message two isn't.

Note, this doesn't do anything smart about the time between those messages or their order. Do you need to take care of that?

0 Karma