Splunk Search

Need help to create alert for a huge number of events

Gene
Path Finder

Dear Splunkers, Hello. I am new to Splunk and have task to create alert for following scenario:

Each minute we receive about 100K events and need to find out events where field value is greater than 180.
Also we have 2 eval fields (current value and previous value)

After each event - current_value = previous_value +(-)1 based on value (greater or less than 180). Also when end of file is reached - next file should start with values of current and previous results.

I have created following search but it doesn't work well:

index=OurIndex

| eval alertType = ""
| eval threshold = 180
| eval severity = "low"
| eval maxLevel = 5
| eval alertLevel = 1
| eval clearLevel = 0


| eval startTime = round(relative_time(_time, "-0s@s"))
| eval processedTime = now()
| eval metric = "dl_dmax"
| eval metricValue = dl_dmax
| streamstats current=f window=1 last(dl_dmax) as lastDmax, last(stateLevel) as lastStateLevel by _time
| eval stateLevel = if(isnull(lastStateLevel), 0, lastStateLevel)
| eval lastLevel = if(lastDmax>threshold, case(stateLevel<maxLevel, stateLevel+1, stateLevel==maxLevel, maxLevel), case(stateLevel!=0, stateLevel-1, stateLevel=0, 0))
| eval stateLevel = if(metricValue>threshold, case(lastLevel<maxLevel, lastLevel+1, lastLevel==maxLevel, maxLevel), case(lastLevel!=0, lastLevel-1, lastLevel=0, 0))
| table snmpid, objectId, objectName, objectType,  alertLevel, lastLevel, stateLevel

 For now stateLevel is never greater than 2 and lastLevel is not greater than 1. Can you please advise on how to modify my search to make it working?

Thanks in advance!

Labels (2)
Tags (2)
0 Karma

Gene
Path Finder

Hello all, I will update my question with example
we have events
id| sessionID | dl_max (numeric value with time to process event)| other not needed fields
num | num| num

what I want to achieve after search

SessionID | dl_max | threshold | lastLevel | stateLevel
1                  | 100  | 180 | 0 | 0

2                  | 190 | 180 | 0 | 1
3                   | 200 | 180 |1 | 2

4                  | 220 | 180 | 2| 3
5                  |160 | 180 |3 | 2

I would really appreciate your help.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...