I'm trying to implement a search query in splunk to get anomalous values around a particular field in the service events.
Essentially the query looks something like this -
index="abc" source=*servicename* response_time | anomalousvalue action=summary pthresh=0.1|search isNum=YES fieldname=response_time
And this gives me a table containing fields like catAnoFreq% , numAnoFreq%, stdev, etc
I looked the documentation https://docs.splunk.com/Documentation/Splunk/9.0.3/SearchReference/Anomalousvalue
but didn't understand how exactly it works.
so for my query if the response_time field has a standard range of values across events, and if my p_thresh=0.1, does that mean that values which occur with a probability of just 10% will fall into the anomalous category? and if i wanted to set an alert on one of the fields in the table to detect anomaly, which would be recommended? i want to set the alert of any event where the response_time num field is not considered within the normal range.
Can anyone help with the follow up question
If p_thresh=0.1, an anomalous event must have at least one field whose value falls below probability of 10% or, if numeric, whose standard deviation is greater than 0.9.
To set alert, it would be simpler to use default action of filter. Something like
index="abc" source=*servicename* response_time
| fields response_time
| anomalousvalue action=filter pthresh=0.1
@yuanliu Thanks.
but if i want to set the alert based on the std value, say if its greater than 30, then using action=summary would be more appropriate?
And i'm assuming its using the gaussian (normal) distribution for the response_time field since useNum=YES?
so if pthresh=0.01 which is 1% , does that mean it will filter the response_time field value which occur below 1%?
I also see some instances of the search returning both useNum=YES and useCat=YES. not sure why that would happen if in that case its still uses the gaussian distribution.