search and alert on a group of consecutive events

[barsuk1]

Hi,
suppose that I have the following log strings:

1616/9  2011-11-22 10:11:23  WARN     program 934478399
1616/9  2011-11-22 10:11:24  WARN     program 934478399
1616/6  2011-11-22 10:11:25  WARN     program 934478399
1616/6  2011-11-22 10:11:26  WARN     program -1
1616/6  2011-11-22 10:11:27  WARN     program -1
1616/9  2011-11-22 10:11:28  WARN     program -1
1616/6  2011-11-22 10:11:29  WARN     program 934478399
1616/6  2011-11-22 10:11:30  WARN     program 934478399
1616/4  2011-11-22 10:11:31  WARN     program 934478399
1616/4  2011-11-22 10:11:32 WARN      program -1
1616/4  2011-11-22 10:11:33 WARN      program 934478399
1616/4  2011-11-22 10:11:34 WARN      program -1
1616/4  2011-11-22 10:11:35 WARN      program -1
1616/4  2011-11-22 10:11:36 WARN      program -1
... same to the end of the log

Now I'd like to find (and alert) on all groups of "program -1" occurences, but only once. They can come by one string, a group of several strings or just start at some moment and continue to the end of the log file. E.g. for the sample above, I'd like to get three matches. I tried transaction but I did not quite got what I wanted.

[carasso]

This removes any duplicate consecutive values, keep only the -1s, and then counts

... | dedup rc consecutive=true | search rc=-1 | stats count

[kristian_kolb]

True indeed. And now I stand corrected, again 🙂

Thanks,

Kristian

[kristian_kolb]

Hi,

Assuming that you have the numerical value (return code?) extracted as field called "rc", you could craft a search like;

...| table rc |delta rc AS d_rc |where d_rc != 0 | stats count(eval(rc=-1)) AS failcount

It does the following:
a) lists the values of the return code
b) adds a column with the difference between the current and previous return code
c) filters out consecutive events with the same return code (i.e. delta=0)
d) count all occurences of rc=-1 and prints them out as "failcount"

This would in your case return the number 3, given your sample events above.

UPDATE: Aah, yes, perhaps I should have been more explicit around that subject. You will get a failcount of n-1, if the log starts with the type of event you are looking for, since the the delta part of the search only detects a change of the rc - whatever is in the first returned event is the initial state.

So to be even more explicit, it is actually NOT if the log starts with that type of event, it's rather if it ENDS with it (since search results are returned newest-event-first by default).

By doing like you did, or piping through reverse, you'll get better results - BUT if your logs start AND end with rc=-1, your counter will be too low by one.

Hope this helps,

Kristian

[kristian_kolb]

see update above

/k

[barsuk1]

search program| sort + _time |delta program AS d_rc |where d_rc != 0 and program = -1 | ...

this actually gives me the three events I wanted

[barsuk1]

ok, thanks. it gives me a direction but it is not 100% correct i.e. I'm getting failcount = 2 on this sample