Splunk Search

How to create alert when error_count is continuously increasing over _time for any group mentioned in column?

StringBee
Explorer

I want to create a alert that will notify if error_count is continuously increasing over time for any of the group mentioned in column
In table I have used timechart which gives sum of error_count value for different groups over the time. I need to compare. I want query that will trigger alert when every row value is greater then its previous row for their respective column, If any column verify this condition Alert should be raised
In Simple words : Alert when error_count increases with time for any group

My sample query:

<<BASE QUERY>> earliest=-4h@h latest=@h | timechart span=30m sum(error_count) as c by group 
Screenshot 2023-02-08 at 3.07.51 PM.png
Result of this query is in image attached ,consider this table as sample data for Alert query

Labels (4)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

It depends on what you mean by "continuously increasing". This might give you a starting point

| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]

StringBee
Explorer

May i know what should be used in <<field>>  because here im working with group and error_count field only, please refer image for more clarity

<<BASE QUERY>> | timechart span=30m sum(error_c) as error_count by group
| streamstats window=1 current=f values(*) as prev_*

| foreach group* [|eval increase_group=if(group > prev_group , 1, null())]

Its just giving each group name and prev_groupname columns
I tried by changing null() by 0 also

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

The foreach command substitutes various special names (including <<FIELD>>) with the fields listed into the subsearch.

foreach - Splunk Documentation

0 Karma

StringBee
Explorer

by "continuously increasing" i mean error_count value should increase consecutively 4 times for respective column
Note : Basically alert should be triggered when we get increasing value in consecutive 4 rows of column

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| streamstats window=1 current=f values(*) as previous_*
| foreach group*
    [| eval increase_<<FIELD>>=if(<<FIELD>> > previous_<<FIELD>>, 1, null())]
| streamstats window=4 sum(increase_*) as last4increase_*
| eval alert=0
| foreach last4increase_*
    [| eval alert=if(alert == 0 and <<FIELD>> == 4, 1, alert)]
| where alert == 1

StringBee
Explorer

You are awesome !
Thanks for help

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Tiling

This puzzle (first published here) is based on finding groups of tessellated tiles (inspired by floor tiles I ...

SOK it to Me: Top 3 Benefits of Using Splunk Operator on Kubernetes that’ll Make ...

    Thursday, July 9, 2026  |  11:00AM–12:00PM PDT Duration: 1 hour (includes Q&A) Managing can feel like a ...

Upgrade Prep for 10.4, Network Observability Deep Dives, and More from Splunk Lantern

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...