Splunk Search

How to create alert when error_count is continuously increasing over _time for any group mentioned in column?

StringBee
Explorer

I want to create a alert that will notify if error_count is continuously increasing over time for any of the group mentioned in column
In table I have used timechart which gives sum of error_count value for different groups over the time. I need to compare. I want query that will trigger alert when every row value is greater then its previous row for their respective column, If any column verify this condition Alert should be raised
In Simple words : Alert when error_count increases with time for any group

My sample query:

<<BASE QUERY>> earliest=-4h@h latest=@h | timechart span=30m sum(error_count) as c by group 
Screenshot 2023-02-08 at 3.07.51 PM.png
Result of this query is in image attached ,consider this table as sample data for Alert query

Labels (4)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you mean by "continuously increasing". This might give you a starting point

| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]

StringBee
Explorer

May i know what should be used in <<field>>  because here im working with group and error_count field only, please refer image for more clarity

<<BASE QUERY>> | timechart span=30m sum(error_c) as error_count by group
| streamstats window=1 current=f values(*) as prev_*

| foreach group* [|eval increase_group=if(group > prev_group , 1, null())]

Its just giving each group name and prev_groupname columns
I tried by changing null() by 0 also

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The foreach command substitutes various special names (including <<FIELD>>) with the fields listed into the subsearch.

foreach - Splunk Documentation

0 Karma

StringBee
Explorer

by "continuously increasing" i mean error_count value should increase consecutively 4 times for respective column
Note : Basically alert should be triggered when we get increasing value in consecutive 4 rows of column

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

StringBee
Explorer

You are awesome !
Thanks for help

0 Karma
Get Updates on the Splunk Community!

Security Professional: Sharpen Your Defenses with These .conf25 Sessions

Sooooooooooo, guess what. .conf25 is almost here, and if you're on the Security Learning Path, this is your ...

First Steps with Splunk SOAR

Our first step was to gather a list of the playbooks we wanted and to sort them by priority.  Once this list ...

How To Build a Self-Service Observability Practice with Splunk Observability Cloud

If you’ve read our previous post on self-service observability, you already know what it is and why it ...