Splunk Search

How to use stats to find abnormal values for an alert?

drodman29
Path Finder

I need an efficient design pattern to move a report that generates a table of values on a 24 hr basis to an alert when these values are statistically "abnormal".

Given a report search like this:

{my search} | eval Loaded=mvzip(key,doc_count)  | mvexpand Loaded  | eval Loaded= split(Loaded,",")  | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1))  | stats   last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by KeyName | eval 24hTotalDiff=MyFirst-Mylast | fields KeyName, 24hTotalDiff

What is the best way to identify abnormal (daily) values of 24hTotalDiff over a xx day sample size?
Should I create a lookup table of average daily values by KeyName? or is there a more elegant solution?

0 Karma
1 Solution

sundareshr
Legend

See if one of these will work for you

{my search} | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | trendline sma2(24hTotalDiff) as Trend | where Trend>=24hTotalDiff*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | table Time KeyName 24hTotalDiff Trend

*OR*

{my search} earliest=-7d@d | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | eventstats stdev(24hTotalDiff) as sd by Key | where 24hTotalDif>=sd*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | ...

View solution in original post

0 Karma

sundareshr
Legend

See if one of these will work for you

{my search} | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | trendline sma2(24hTotalDiff) as Trend | where Trend>=24hTotalDiff*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | table Time KeyName 24hTotalDiff Trend

*OR*

{my search} earliest=-7d@d | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) | bin span=1d _time as t | eval Key=t."#".KeyName | stats last(Doc_Count) as Mylast first(Doc_Count) as MyFirst by Key | eval 24hTotalDiff=MyFirst-Mylast | eventstats stdev(24hTotalDiff) as sd by Key | where 24hTotalDif>=sd*2 | rex field=Key "(?<Time>[^#]+)#(?<KeyName>.*)" | ...
0 Karma

drodman29
Path Finder

The design pattern seems to be "concatenate a time value to the Stats by-clause variable that is consistent with the time based aggregation that you want, then separate the compound key and run eventstats to get sd and avg by the original key."

Posting corrected answer syntax.

{mySearch} | eval Loaded=mvzip(key,doc_count) | mvexpand Loaded | eval Loaded= split(Loaded,",") | eval KeyName=mvindex(Loaded,0) | eval Doc_Count=tonumber(mvindex(Loaded,1)) |  search KeyName!="AFI*" | eval MasterKey=strftime(_time,"%Y-%m-%d" )."#".KeyName | stats latest(Doc_Count) as MyLast earliest(Doc_Count) as MyFirst by MasterKey | eval TotalDiff=tonumber(MyLast-MyFirst)| rex field=MasterKey "(?<YearDate>[^#]+)#(?<KeyName>.*)" | eventstats stdev(TotalDiff) as SD avg(TotalDiff) as AVG by KeyName | table MasterKey, KeyName, TotalDiff, SD , AVG | eval KeyStatus=if(abs(TotalDiff -AVG ) >  SD*2, "RED", "GREEN") | fields MasterKey, KeyName, TotalDiff, SD , AVG, KeyStatus
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...