Conditional searches

[efo]

Hi,
We have gotten quite a complex search request, which we are not sure if is possible at all.

If the application log says "SITE X DOWN", and then within an five minute interval logs "SITE BACKUP X DOWN" it must raise an alert. If the applications logs SITE BAKCUP X DOWN outside the five minute interval everything is OK.

How, if at all possible, could this be accomplished?

--
Espen

[efo]

This is perfect, thank you.
But is it best run with time range "rt-5 to rt" or just a normal search that ranges the last ten minutes or so?

[Ayn · Accepted solution]

You can use the transaction command. This will join separate events together to a new combined event (a transaction) based on rules that you specify. You can then search for transactions that match multiple conditions.

In your case, you want to find cases where "SITE X DOWN" and "SITE BACKUP X DOWN" occur within a 5 minute interval. This can be accomplished using transaction like this:

"SITE X DOWN" OR "SITE BACKUP X DOWN" 
| transaction source startswith="SITE X DOWN" endswith="SITE BACKUP X DOWN" maxspan=5m

Any events returned by this search will match your condition. I used "source" as an argument to transaction but any field identifier can be used. This specifies what field(s) Splunk should look for and use when grouping together events, so in this case Splunk will be looking to grouping events into transactions if they have the same value for the "source" field. The more unique the field value, the better.

More information on the transaction command is available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction

[Ayn]

Just don't use source as a field to run transaction on.

[lpolo]

How would you do it if you have two different source logs?

[efo]

This solves the problem.
Thank you very much, Ayn

[Drainy]

Excellent answer, clearly explained