Splunk Search

search and alert on a group of consecutive events

barsuk1
New Member

Hi,
suppose that I have the following log strings:

1616/9  2011-11-22 10:11:23  WARN     program 934478399
1616/9  2011-11-22 10:11:24  WARN     program 934478399
1616/6  2011-11-22 10:11:25  WARN     program 934478399
1616/6  2011-11-22 10:11:26  WARN     program -1
1616/6  2011-11-22 10:11:27  WARN     program -1
1616/9  2011-11-22 10:11:28  WARN     program -1
1616/6  2011-11-22 10:11:29  WARN     program 934478399
1616/6  2011-11-22 10:11:30  WARN     program 934478399
1616/4  2011-11-22 10:11:31  WARN     program 934478399
1616/4  2011-11-22 10:11:32 WARN      program -1
1616/4  2011-11-22 10:11:33 WARN      program 934478399
1616/4  2011-11-22 10:11:34 WARN      program -1
1616/4  2011-11-22 10:11:35 WARN      program -1
1616/4  2011-11-22 10:11:36 WARN      program -1
... same to the end of the log

Now I'd like to find (and alert) on all groups of "program -1" occurences, but only once. They can come by one string, a group of several strings or just start at some moment and continue to the end of the log file. E.g. for the sample above, I'd like to get three matches. I tried transaction but I did not quite got what I wanted.

Tags (5)
0 Karma

carasso
Splunk Employee
Splunk Employee

This removes any duplicate consecutive values, keep only the -1s, and then counts

... | dedup rc consecutive=true | search rc=-1 | stats count

kristian_kolb
Ultra Champion

True indeed. And now I stand corrected, again 🙂

Thanks,

Kristian

0 Karma

kristian_kolb
Ultra Champion

Hi,

Assuming that you have the numerical value (return code?) extracted as field called "rc", you could craft a search like;

...| table rc |delta rc AS d_rc |where d_rc != 0 | stats count(eval(rc=-1)) AS failcount

It does the following:
a) lists the values of the return code
b) adds a column with the difference between the current and previous return code
c) filters out consecutive events with the same return code (i.e. delta=0)
d) count all occurences of rc=-1 and prints them out as "failcount"

This would in your case return the number 3, given your sample events above.

UPDATE: Aah, yes, perhaps I should have been more explicit around that subject. You will get a failcount of n-1, if the log starts with the type of event you are looking for, since the the delta part of the search only detects a change of the rc - whatever is in the first returned event is the initial state.

So to be even more explicit, it is actually NOT if the log starts with that type of event, it's rather if it ENDS with it (since search results are returned newest-event-first by default).

By doing like you did, or piping through reverse, you'll get better results - BUT if your logs start AND end with rc=-1, your counter will be too low by one.

Hope this helps,

Kristian

kristian_kolb
Ultra Champion

see update above

/k

0 Karma

barsuk1
New Member

search program| sort + _time |delta program AS d_rc |where d_rc != 0 and program = -1 | ...

this actually gives me the three events I wanted

0 Karma

barsuk1
New Member

ok, thanks. it gives me a direction but it is not 100% correct i.e. I'm getting failcount = 2 on this sample

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...