Splunk Search

Show triggered events from last 5 minutes from a 2 hour moving average

Habanero
Explorer

Good day,

We are looking at a solution to alert us on abnormal traffic spike. We have leverage the standard deviation, and `streamstats` for the moving average. We are "graphing" for the last 2 hours. Last but not least, there is a cron job running every 2 minutes. Below is the query:

 

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") AS avg stdev("count") AS stdev
| eval upperBound=(avg+stdev*exact(2))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) | rename count as "Events" upperBound AS"Upper Limit" isOutlier AS"Is Outlier" avg AS "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"
| search "Is Outlier"=1

 

 

The problem I am encountering is once there is a "Outlier" it will remain in the table for the next 2 hours.
i.e. Outlier a 7:31am on the next schedule run at 7:32am it will trigger. But the entry will still show up at 7:34am, 7:36am, and so forth.

I tried using the following arguments but it doesn't work.

 

| search "Is Outlier"=1 earliest=-2m@m latest=now()

 


Does anyone has any idea how I can have the alerts show the last two minutes, but retaining the 2 hours moving average?

Thank you in advance!

 

Labels (4)
0 Karma
1 Solution

Habanero
Explorer

I was able to figure out a solution to my problem.

Here is the complete query:

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") as avg stdev("count") as stdev
| eval upperBound=(avg+stdev*exact(1))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) 
| eval last_three_mins=relative_time(now(), "-3m@m")
| search "isOutlier"=1
| where _time >= last_three_mins
| rename count as "Events" upperBound as "Upper Limit" isOutlier as "Is Outlier" avg as "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"

 

Thanks to everyone who helped.

 

View solution in original post

0 Karma

Habanero
Explorer

I was able to figure out a solution to my problem.

Here is the complete query:

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") as avg stdev("count") as stdev
| eval upperBound=(avg+stdev*exact(1))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) 
| eval last_three_mins=relative_time(now(), "-3m@m")
| search "isOutlier"=1
| where _time >= last_three_mins
| rename count as "Events" upperBound as "Upper Limit" isOutlier as "Is Outlier" avg as "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"

 

Thanks to everyone who helped.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

Try this search

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") AS avg stdev("count") AS stdev
| eval upperBound=(avg+stdev*exact(2))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) | rename count as "Events" upperBound AS"Upper Limit" isOutlier AS"Is Outlier" avg AS "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"
| where ("Is Outlier"=1 AND _time >= relative_time(now(), "-2m"))
---
If this reply helps you, Karma would be appreciated.
0 Karma

Habanero
Explorer

Thanks for replying, although your search gives me this error:

Error in 'where' command: Type checking failed. The '==' operator received different types

 

 

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...