Splunk Search

How to create alert when error_count is continuously increasing over _time for any group mentioned in column?

StringBee
Explorer

I want to create a alert that will notify if error_count is continuously increasing over time for any of the group mentioned in column
In table I have used timechart which gives sum of error_count value for different groups over the time. I need to compare. I want query that will trigger alert when every row value is greater then its previous row for their respective column, If any column verify this condition Alert should be raised
In Simple words : Alert when error_count increases with time for any group

My sample query:

<<BASE QUERY>> earliest=-4h@h latest=@h | timechart span=30m sum(error_count) as c by group 
Screenshot 2023-02-08 at 3.07.51 PM.png
Result of this query is in image attached ,consider this table as sample data for Alert query

Labels (4)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you mean by "continuously increasing". This might give you a starting point

| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]

StringBee
Explorer

May i know what should be used in <<field>>  because here im working with group and error_count field only, please refer image for more clarity

<<BASE QUERY>> | timechart span=30m sum(error_c) as error_count by group
| streamstats window=1 current=f values(*) as prev_*

| foreach group* [|eval increase_group=if(group > prev_group , 1, null())]

Its just giving each group name and prev_groupname columns
I tried by changing null() by 0 also

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The foreach command substitutes various special names (including <<FIELD>>) with the fields listed into the subsearch.

foreach - Splunk Documentation

0 Karma

StringBee
Explorer

by "continuously increasing" i mean error_count value should increase consecutively 4 times for respective column
Note : Basically alert should be triggered when we get increasing value in consecutive 4 rows of column

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

StringBee
Explorer

You are awesome !
Thanks for help

0 Karma
Get Updates on the Splunk Community!

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...

Cloud Monitoring Console - Unlocking Greater Visibility in SVC Usage Reporting

For Splunk Cloud customers, understanding and optimizing Splunk Virtual Compute (SVC) usage and resource ...

Automatic Discovery Part 3: Practical Use Cases

If you’ve enabled Automatic Discovery in your install of the Splunk Distribution of the OpenTelemetry ...