Splunk Search

Show triggered events from last 5 minutes from a 2 hour moving average

Habanero
Explorer

Good day,

We are looking at a solution to alert us on abnormal traffic spike. We have leverage the standard deviation, and `streamstats` for the moving average. We are "graphing" for the last 2 hours. Last but not least, there is a cron job running every 2 minutes. Below is the query:

 

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") AS avg stdev("count") AS stdev
| eval upperBound=(avg+stdev*exact(2))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) | rename count as "Events" upperBound AS"Upper Limit" isOutlier AS"Is Outlier" avg AS "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"
| search "Is Outlier"=1

 

 

The problem I am encountering is once there is a "Outlier" it will remain in the table for the next 2 hours.
i.e. Outlier a 7:31am on the next schedule run at 7:32am it will trigger. But the entry will still show up at 7:34am, 7:36am, and so forth.

I tried using the following arguments but it doesn't work.

 

| search "Is Outlier"=1 earliest=-2m@m latest=now()

 


Does anyone has any idea how I can have the alerts show the last two minutes, but retaining the 2 hours moving average?

Thank you in advance!

 

Labels (5)
0 Karma
1 Solution

Habanero
Explorer

I was able to figure out a solution to my problem.

Here is the complete query:

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") as avg stdev("count") as stdev
| eval upperBound=(avg+stdev*exact(1))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) 
| eval last_three_mins=relative_time(now(), "-3m@m")
| search "isOutlier"=1
| where _time >= last_three_mins
| rename count as "Events" upperBound as "Upper Limit" isOutlier as "Is Outlier" avg as "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"

 

Thanks to everyone who helped.

 

View solution in original post

0 Karma

Habanero
Explorer

I was able to figure out a solution to my problem.

Here is the complete query:

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") as avg stdev("count") as stdev
| eval upperBound=(avg+stdev*exact(1))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) 
| eval last_three_mins=relative_time(now(), "-3m@m")
| search "isOutlier"=1
| where _time >= last_three_mins
| rename count as "Events" upperBound as "Upper Limit" isOutlier as "Is Outlier" avg as "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"

 

Thanks to everyone who helped.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If your problem is resolved, then please click the "Accept as Solution" button to help future readers.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

Try this search

base_search earliest=-121m@m latest=-1m@m | bin _time span=2m | stats count by _time
| streamstats avg("count") AS avg stdev("count") AS stdev
| eval upperBound=(avg+stdev*exact(2))
| eval isOutlier=if('count' < lowerBound OR 'count' > upperBound, 1, 0)
| eval avg=round(avg,0) | eval upperBound=round(upperBound,0) | rename count as "Events" upperBound AS"Upper Limit" isOutlier AS"Is Outlier" avg AS "Average" 
| fields _time, "Events", "Average", "Upper Limit", "Is Outlier"
| where ("Is Outlier"=1 AND _time >= relative_time(now(), "-2m"))
---
If this reply helps you, Karma would be appreciated.
0 Karma

Habanero
Explorer

Thanks for replying, although your search gives me this error:

Error in 'where' command: Type checking failed. The '==' operator received different types

 

 

0 Karma
Get Updates on the Splunk Community!

Changes to Splunk Instructor-Led Training Completion Criteria

We’re excited to share an update to our instructor-led training program that enhances the learning experience ...

Stay Connected: Your Guide to January Tech Talks, Office Hours, and Webinars!

❄️ Welcome the new year with our January lineup of Community Office Hours, Tech Talks, and Webinars! &#x1f389; ...

Preparing your Splunk Environment for OpenSSL3

The Splunk platform will transition to OpenSSL version 3 in a future release. Actions are required to prepare ...