Splunk Search

How to create alert when error_count is continuously increasing over _time for any group mentioned in column?

StringBee
Explorer

I want to create a alert that will notify if error_count is continuously increasing over time for any of the group mentioned in column
In table I have used timechart which gives sum of error_count value for different groups over the time. I need to compare. I want query that will trigger alert when every row value is greater then its previous row for their respective column, If any column verify this condition Alert should be raised
In Simple words : Alert when error_count increases with time for any group

My sample query:

<<BASE QUERY>> earliest=-4h@h latest=@h | timechart span=30m sum(error_count) as c by group 
Screenshot 2023-02-08 at 3.07.51 PM.png
Result of this query is in image attached ,consider this table as sample data for Alert query

Labels (4)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you mean by "continuously increasing". This might give you a starting point

| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]

StringBee
Explorer

May i know what should be used in <<field>>  because here im working with group and error_count field only, please refer image for more clarity

<<BASE QUERY>> | timechart span=30m sum(error_c) as error_count by group
| streamstats window=1 current=f values(*) as prev_*

| foreach group* [|eval increase_group=if(group > prev_group , 1, null())]

Its just giving each group name and prev_groupname columns
I tried by changing null() by 0 also

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The foreach command substitutes various special names (including <<FIELD>>) with the fields listed into the subsearch.

foreach - Splunk Documentation

0 Karma

StringBee
Explorer

by "continuously increasing" i mean error_count value should increase consecutively 4 times for respective column
Note : Basically alert should be triggered when we get increasing value in consecutive 4 rows of column

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

StringBee
Explorer

You are awesome !
Thanks for help

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...