Splunk Search

search and alert on a group of consecutive events

barsuk1
New Member

Hi,
suppose that I have the following log strings:

1616/9  2011-11-22 10:11:23  WARN     program 934478399
1616/9  2011-11-22 10:11:24  WARN     program 934478399
1616/6  2011-11-22 10:11:25  WARN     program 934478399
1616/6  2011-11-22 10:11:26  WARN     program -1
1616/6  2011-11-22 10:11:27  WARN     program -1
1616/9  2011-11-22 10:11:28  WARN     program -1
1616/6  2011-11-22 10:11:29  WARN     program 934478399
1616/6  2011-11-22 10:11:30  WARN     program 934478399
1616/4  2011-11-22 10:11:31  WARN     program 934478399
1616/4  2011-11-22 10:11:32 WARN      program -1
1616/4  2011-11-22 10:11:33 WARN      program 934478399
1616/4  2011-11-22 10:11:34 WARN      program -1
1616/4  2011-11-22 10:11:35 WARN      program -1
1616/4  2011-11-22 10:11:36 WARN      program -1
... same to the end of the log

Now I'd like to find (and alert) on all groups of "program -1" occurences, but only once. They can come by one string, a group of several strings or just start at some moment and continue to the end of the log file. E.g. for the sample above, I'd like to get three matches. I tried transaction but I did not quite got what I wanted.

Tags (5)
0 Karma

carasso
Splunk Employee
Splunk Employee

This removes any duplicate consecutive values, keep only the -1s, and then counts

... | dedup rc consecutive=true | search rc=-1 | stats count

kristian_kolb
Ultra Champion

True indeed. And now I stand corrected, again 🙂

Thanks,

Kristian

0 Karma

kristian_kolb
Ultra Champion

Hi,

Assuming that you have the numerical value (return code?) extracted as field called "rc", you could craft a search like;

...| table rc |delta rc AS d_rc |where d_rc != 0 | stats count(eval(rc=-1)) AS failcount

It does the following:
a) lists the values of the return code
b) adds a column with the difference between the current and previous return code
c) filters out consecutive events with the same return code (i.e. delta=0)
d) count all occurences of rc=-1 and prints them out as "failcount"

This would in your case return the number 3, given your sample events above.

UPDATE: Aah, yes, perhaps I should have been more explicit around that subject. You will get a failcount of n-1, if the log starts with the type of event you are looking for, since the the delta part of the search only detects a change of the rc - whatever is in the first returned event is the initial state.

So to be even more explicit, it is actually NOT if the log starts with that type of event, it's rather if it ENDS with it (since search results are returned newest-event-first by default).

By doing like you did, or piping through reverse, you'll get better results - BUT if your logs start AND end with rc=-1, your counter will be too low by one.

Hope this helps,

Kristian

kristian_kolb
Ultra Champion

see update above

/k

0 Karma

barsuk1
New Member

search program| sort + _time |delta program AS d_rc |where d_rc != 0 and program = -1 | ...

this actually gives me the three events I wanted

0 Karma

barsuk1
New Member

ok, thanks. it gives me a direction but it is not 100% correct i.e. I'm getting failcount = 2 on this sample

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!