Splunk Search

Calculate difference in event counts and trigger alert if > 1

Engager

Hello,

I am trying to devise a search that will basically count the number of two different log statements, and then from that search alert if their difference is > 1.

e.g. the search that returns the list of events is "Sending msg" OR "Received msg". I want to basically count the number of times "Sending msg" is found as one count, "Received msg" count as another, and alert if count( "Sending msg" ) - count( "Received msg" ) > 0 over a 5 minute window.

However there aren't any fields that I can use to count, it's just a free-text log statement. I'm struggling with the splunk syntax to accomplish this.

Any advice/help is appreciated.

Thanks,
Ryan

Tags (3)
1 Solution

Motivator

Without some sample data it's hard to tell what the best way to accomplish this would be, however I assume your events look like this:

[2011-05-05 12:12:12] Sending msg
[2011-05-05 12:13:10] Received msg

So you could create a search like the following:

[your search terms] | rex "(?<action>Sending|Received)" | stats count(eval(action="Sending")) AS SendingCount, count(eval(action="Receiving")) AS ReceivingCount | eval difference=SendingCount-ReceivingCount

And then when you schedule your search you can specify an advanced alert condition:

search difference > 0

View solution in original post

Motivator

Without some sample data it's hard to tell what the best way to accomplish this would be, however I assume your events look like this:

[2011-05-05 12:12:12] Sending msg
[2011-05-05 12:13:10] Received msg

So you could create a search like the following:

[your search terms] | rex "(?<action>Sending|Received)" | stats count(eval(action="Sending")) AS SendingCount, count(eval(action="Receiving")) AS ReceivingCount | eval difference=SendingCount-ReceivingCount

And then when you schedule your search you can specify an advanced alert condition:

search difference > 0

View solution in original post

Engager

That is exactly what I'm after. Thanks. Sorry I didn't include the log formats in my original post, but your inference was correct. I'm also reformatting these log statements so that I can use the transaction function to identify the individual transactions. Basically adding an identifier to correlate the sending & receiving log events. But this is an excellent first step. Thanks!

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!