Splunk Search

Detecting outlier based on result of timechart



I am trying to detect if any of the server in farm decrease in performance. I can see performance going down as the number of logs go down by using the following query:

index=xxx | timechart span=5m count as event_count by host

Plotting into a graph and I can see one of the lines going down and I can say the performance goes down.
I am trying to use the results of the query above to find any host event count is below 2 stdev for past 10 minutes, but doesn't seem like I can use event_count for subsequence pipes. Is there anyway to achieve this?

0 Karma

Ultra Champion
| makeresults count=2
| streamstats count
| eval _time = if (count==2,relative_time(_time,"-2h@m"), relative_time(_time,"@m")) 
| makecontinuous span=15s _time
| eval host="host".(random() % 5 + 1)
| timechart span=5m count as event_count by host
| rename COMMENT as "this is sample you provide"
| rename COMMENT as "from here, the logic"
| eval tmp=""
| foreach host*
    [eval tmp=mvappend(tmp,<<FIELD>>)
     | eval _n=mvcount(tmp) - 1]
| table _* host*
| streamstats current=f window=2 stdev(*) as *_past10min_stdv
| foreach host*_past10min_stdv
    [eval <<FIELD>>2= 2 * '<<FIELD>>']
| reverse
| rename _n as n

Hi, @dyuen
I did the calculation.
However, since the data is a random number, I can't get a proper number.
please provide sample.

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! &#x1f308; In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...