Splunk Search

Need help to create alert for a huge number of events

Gene
Path Finder

Dear Splunkers, Hello. I am new to Splunk and have task to create alert for following scenario:

Each minute we receive about 100K events and need to find out events where field value is greater than 180.
Also we have 2 eval fields (current value and previous value)

After each event - current_value = previous_value +(-)1 based on value (greater or less than 180). Also when end of file is reached - next file should start with values of current and previous results.

I have created following search but it doesn't work well:

index=OurIndex

| eval alertType = ""
| eval threshold = 180
| eval severity = "low"
| eval maxLevel = 5
| eval alertLevel = 1
| eval clearLevel = 0


| eval startTime = round(relative_time(_time, "-0s@s"))
| eval processedTime = now()
| eval metric = "dl_dmax"
| eval metricValue = dl_dmax
| streamstats current=f window=1 last(dl_dmax) as lastDmax, last(stateLevel) as lastStateLevel by _time
| eval stateLevel = if(isnull(lastStateLevel), 0, lastStateLevel)
| eval lastLevel = if(lastDmax>threshold, case(stateLevel<maxLevel, stateLevel+1, stateLevel==maxLevel, maxLevel), case(stateLevel!=0, stateLevel-1, stateLevel=0, 0))
| eval stateLevel = if(metricValue>threshold, case(lastLevel<maxLevel, lastLevel+1, lastLevel==maxLevel, maxLevel), case(lastLevel!=0, lastLevel-1, lastLevel=0, 0))
| table snmpid, objectId, objectName, objectType,  alertLevel, lastLevel, stateLevel

 For now stateLevel is never greater than 2 and lastLevel is not greater than 1. Can you please advise on how to modify my search to make it working?

Thanks in advance!

Labels (2)
Tags (2)
0 Karma

Gene
Path Finder

Hello all, I will update my question with example
we have events
id| sessionID | dl_max (numeric value with time to process event)| other not needed fields
num | num| num

what I want to achieve after search

SessionID | dl_max | threshold | lastLevel | stateLevel
1                  | 100  | 180 | 0 | 0

2                  | 190 | 180 | 0 | 1
3                   | 200 | 180 |1 | 2

4                  | 220 | 180 | 2| 3
5                  |160 | 180 |3 | 2

I would really appreciate your help.

0 Karma
Get Updates on the Splunk Community!

CX Day is Coming!

Customer Experience (CX) Day is on October 7th!! We're so excited to bring back another day full of wonderful ...

Strengthen Your Future: A Look Back at Splunk 10 Innovations and .conf25 Highlights!

The Big One: Splunk 10 is Here!  The moment many of you have been waiting for has arrived! We are thrilled to ...

Now Offering the AI Assistant Usage Dashboard in Cloud Monitoring Console

Today, we’re excited to announce the release of a brand new AI assistant usage dashboard in Cloud Monitoring ...