Splunk Search

Need help to create alert for a huge number of events

Gene
Path Finder

Dear Splunkers, Hello. I am new to Splunk and have task to create alert for following scenario:

Each minute we receive about 100K events and need to find out events where field value is greater than 180.
Also we have 2 eval fields (current value and previous value)

After each event - current_value = previous_value +(-)1 based on value (greater or less than 180). Also when end of file is reached - next file should start with values of current and previous results.

I have created following search but it doesn't work well:

index=OurIndex

| eval alertType = ""
| eval threshold = 180
| eval severity = "low"
| eval maxLevel = 5
| eval alertLevel = 1
| eval clearLevel = 0


| eval startTime = round(relative_time(_time, "-0s@s"))
| eval processedTime = now()
| eval metric = "dl_dmax"
| eval metricValue = dl_dmax
| streamstats current=f window=1 last(dl_dmax) as lastDmax, last(stateLevel) as lastStateLevel by _time
| eval stateLevel = if(isnull(lastStateLevel), 0, lastStateLevel)
| eval lastLevel = if(lastDmax>threshold, case(stateLevel<maxLevel, stateLevel+1, stateLevel==maxLevel, maxLevel), case(stateLevel!=0, stateLevel-1, stateLevel=0, 0))
| eval stateLevel = if(metricValue>threshold, case(lastLevel<maxLevel, lastLevel+1, lastLevel==maxLevel, maxLevel), case(lastLevel!=0, lastLevel-1, lastLevel=0, 0))
| table snmpid, objectId, objectName, objectType,  alertLevel, lastLevel, stateLevel

 For now stateLevel is never greater than 2 and lastLevel is not greater than 1. Can you please advise on how to modify my search to make it working?

Thanks in advance!

Labels (2)
Tags (2)
0 Karma

Gene
Path Finder

Hello all, I will update my question with example
we have events
id| sessionID | dl_max (numeric value with time to process event)| other not needed fields
num | num| num

what I want to achieve after search

SessionID | dl_max | threshold | lastLevel | stateLevel
1                  | 100  | 180 | 0 | 0

2                  | 190 | 180 | 0 | 1
3                   | 200 | 180 |1 | 2

4                  | 220 | 180 | 2| 3
5                  |160 | 180 |3 | 2

I would really appreciate your help.

0 Karma
Get Updates on the Splunk Community!

Index This | What’s a riddle wrapped in an enigma?

September 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

BORE at .conf25

Boss Of Regular Expression (BORE) was an interactive session run again this year at .conf25 by the brilliant ...

OpenTelemetry for Legacy Apps? Yes, You Can!

This article is a follow-up to my previous article posted on the OpenTelemetry Blog, "Your Critical Legacy App ...