Splunk Search

Detecting outlier based on result of timechart



I am trying to detect if any of the server in farm decrease in performance. I can see performance going down as the number of logs go down by using the following query:

index=xxx | timechart span=5m count as event_count by host

Plotting into a graph and I can see one of the lines going down and I can say the performance goes down.
I am trying to use the results of the query above to find any host event count is below 2 stdev for past 10 minutes, but doesn't seem like I can use event_count for subsequence pipes. Is there anyway to achieve this?

0 Karma

Ultra Champion
| makeresults count=2
| streamstats count
| eval _time = if (count==2,relative_time(_time,"-2h@m"), relative_time(_time,"@m")) 
| makecontinuous span=15s _time
| eval host="host".(random() % 5 + 1)
| timechart span=5m count as event_count by host
| rename COMMENT as "this is sample you provide"
| rename COMMENT as "from here, the logic"
| eval tmp=""
| foreach host*
    [eval tmp=mvappend(tmp,<<FIELD>>)
     | eval _n=mvcount(tmp) - 1]
| table _* host*
| streamstats current=f window=2 stdev(*) as *_past10min_stdv
| foreach host*_past10min_stdv
    [eval <<FIELD>>2= 2 * '<<FIELD>>']
| reverse
| rename _n as n

Hi, @dyuen
I did the calculation.
However, since the data is a random number, I can't get a proper number.
please provide sample.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...