Splunk Search

Need help to create alert for a huge number of events

Gene
Path Finder

Dear Splunkers, Hello. I am new to Splunk and have task to create alert for following scenario:

Each minute we receive about 100K events and need to find out events where field value is greater than 180.
Also we have 2 eval fields (current value and previous value)

After each event - current_value = previous_value +(-)1 based on value (greater or less than 180). Also when end of file is reached - next file should start with values of current and previous results.

I have created following search but it doesn't work well:

index=OurIndex

| eval alertType = ""
| eval threshold = 180
| eval severity = "low"
| eval maxLevel = 5
| eval alertLevel = 1
| eval clearLevel = 0


| eval startTime = round(relative_time(_time, "-0s@s"))
| eval processedTime = now()
| eval metric = "dl_dmax"
| eval metricValue = dl_dmax
| streamstats current=f window=1 last(dl_dmax) as lastDmax, last(stateLevel) as lastStateLevel by _time
| eval stateLevel = if(isnull(lastStateLevel), 0, lastStateLevel)
| eval lastLevel = if(lastDmax>threshold, case(stateLevel<maxLevel, stateLevel+1, stateLevel==maxLevel, maxLevel), case(stateLevel!=0, stateLevel-1, stateLevel=0, 0))
| eval stateLevel = if(metricValue>threshold, case(lastLevel<maxLevel, lastLevel+1, lastLevel==maxLevel, maxLevel), case(lastLevel!=0, lastLevel-1, lastLevel=0, 0))
| table snmpid, objectId, objectName, objectType,  alertLevel, lastLevel, stateLevel

 For now stateLevel is never greater than 2 and lastLevel is not greater than 1. Can you please advise on how to modify my search to make it working?

Thanks in advance!

Labels (2)
Tags (2)
0 Karma

Gene
Path Finder

Hello all, I will update my question with example
we have events
id| sessionID | dl_max (numeric value with time to process event)| other not needed fields
num | num| num

what I want to achieve after search

SessionID | dl_max | threshold | lastLevel | stateLevel
1                  | 100  | 180 | 0 | 0

2                  | 190 | 180 | 0 | 1
3                   | 200 | 180 |1 | 2

4                  | 220 | 180 | 2| 3
5                  |160 | 180 |3 | 2

I would really appreciate your help.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In January, the Splunk Threat Research Team had one release of new security content via the Splunk ES Content ...

Expert Tips from Splunk Professional Services, Ensuring Compliance, and More New ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Observability Release Update: AI Assistant, AppD + Observability Cloud Integrations & ...

This month’s releases across the Splunk Observability portfolio deliver earlier detection and faster ...