Splunk Search

How to create alert when error_count is continuously increasing over _time for any group mentioned in column?

StringBee
Explorer

I want to create a alert that will notify if error_count is continuously increasing over time for any of the group mentioned in column
In table I have used timechart which gives sum of error_count value for different groups over the time. I need to compare. I want query that will trigger alert when every row value is greater then its previous row for their respective column, If any column verify this condition Alert should be raised
In Simple words : Alert when error_count increases with time for any group

My sample query:

<<BASE QUERY>> earliest=-4h@h latest=@h | timechart span=30m sum(error_count) as c by group 
Screenshot 2023-02-08 at 3.07.51 PM.png
Result of this query is in image attached ,consider this table as sample data for Alert query

Labels (4)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you mean by "continuously increasing". This might give you a starting point

| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]

StringBee
Explorer

May i know what should be used in <<field>>  because here im working with group and error_count field only, please refer image for more clarity

<<BASE QUERY>> | timechart span=30m sum(error_c) as error_count by group
| streamstats window=1 current=f values(*) as prev_*

| foreach group* [|eval increase_group=if(group > prev_group , 1, null())]

Its just giving each group name and prev_groupname columns
I tried by changing null() by 0 also

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The foreach command substitutes various special names (including <<FIELD>>) with the fields listed into the subsearch.

foreach - Splunk Documentation

0 Karma

StringBee
Explorer

by "continuously increasing" i mean error_count value should increase consecutively 4 times for respective column
Note : Basically alert should be triggered when we get increasing value in consecutive 4 rows of column

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

StringBee
Explorer

You are awesome !
Thanks for help

0 Karma
Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

Splunk Developers: Go Beyond the Dashboard with These .Conf25 Sessions

  Whether you’re building custom apps, diving into SPL2, or integrating AI and machine learning into your ...

Index This | How do you write 23 only using the number 2?

July 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...