topic Re: Conditional searches in Splunk Search

Conditional searches

efo — Wed, 31 Aug 2011 07:54:12 GMT

Hi,
We have gotten quite a complex search request, which we are not sure if is possible at all.

If the application log says "SITE X DOWN", and then within an five minute interval logs "SITE BACKUP X DOWN" it must raise an alert. If the applications logs SITE BAKCUP X DOWN outside the five minute interval everything is OK.

How, if at all possible, could this be accomplished?

--
Espen

Re: Conditional searches

Ayn — Wed, 31 Aug 2011 08:12:43 GMT

You can use the transaction command. This will join separate events together to a new combined event (a transaction) based on rules that you specify. You can then search for transactions that match multiple conditions.

In your case, you want to find cases where "SITE X DOWN" and "SITE BACKUP X DOWN" occur within a 5 minute interval. This can be accomplished using transaction like this:

"SITE X DOWN" OR "SITE BACKUP X DOWN" 
| transaction source startswith="SITE X DOWN" endswith="SITE BACKUP X DOWN" maxspan=5m

Any events returned by this search will match your condition. I used "source" as an argument to transaction but any field identifier can be used. This specifies what field(s) Splunk should look for and use when grouping together events, so in this case Splunk will be looking to grouping events into transactions if they have the same value for the "source" field. The more unique the field value, the better.

More information on the transaction command is available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction

Re: Conditional searches

Drainy — Wed, 31 Aug 2011 08:14:35 GMT

Excellent answer, clearly explained

Re: Conditional searches

efo — Thu, 15 Sep 2011 06:58:35 GMT

This solves the problem.
Thank you very much, Ayn

Re: Conditional searches

efo — Thu, 15 Sep 2011 07:25:03 GMT

This is perfect, thank you.
But is it best run with time range "rt-5 to rt" or just a normal search that ranges the last ten minutes or so?

Re: Conditional searches

lpolo — Thu, 11 Oct 2012 12:09:33 GMT

How would you do it if you have two different source logs?

Re: Conditional searches

Ayn — Thu, 11 Oct 2012 12:13:37 GMT

Just don't use source as a field to run transaction on.