Dashboards & Visualizations

Discrepancies between MLTK and Splunk App for Anomaly Detection

danielbb
Motivator

With MLTK, when looking at accumulated runtime, the outliers are detected cleanly (three out of three spikes), whereas with the anomaly detection app, only two of the three spikes are detected (along with one false positive, even at medium sensitivity).

mltk_median_runtime_comparison.PNG

The code generated by the MLTK is as follows -

 

index=_audit host=XXXXXXXX action=search info=completed 
| table _time host total_run_time savedsearch_name 
| eval total_run_time_mins=total_run_time/60 
| convert ctime(search_*) 
| eval savedsearch_name=if(savedsearch_name="","Ad-hoc",savedsearch_name) 
| search savedsearch_name!="_ACCEL*" AND savedsearch_name!="Ad-hoc" 
| timechart span=30m median(total_run_time_mins)

| eval "atf_hour_of_day"=strftime(_time, "%H"), "atf_day_of_week"=strftime(_time, "%w-%A"), "atf_day_of_month"=strftime(_time, "%e"), "atf_month" = strftime(_time, "%m-%B") 
| eventstats dc("atf_hour_of_day"),dc("atf_day_of_week"),dc("atf_day_of_month"),dc("atf_month") | eval "atf_hour_of_day"=if('dc(atf_hour_of_day)'<2, null(), 'atf_hour_of_day'),"atf_day_of_week"=if('dc(atf_day_of_week)'<2, null(), 'atf_day_of_week'),"atf_day_of_month"=if('dc(atf_day_of_month)'<2, null(), 'atf_day_of_month'),"atf_month"=if('dc(atf_month)'<2, null(), 'atf_month') | fields - "dc(atf_hour_of_day)","dc(atf_day_of_week)","dc(atf_day_of_month)","dc(atf_month)" | eval "_atf_hour_of_day_copy"=atf_hour_of_day,"_atf_day_of_week_copy"=atf_day_of_week,"_atf_day_of_month_copy"=atf_day_of_month,"_atf_month_copy"=atf_month | fields - "atf_hour_of_day","atf_day_of_week","atf_day_of_month","atf_month" | rename "_atf_hour_of_day_copy" as "atf_hour_of_day","_atf_day_of_week_copy" as "atf_day_of_week","_atf_day_of_month_copy" as "atf_day_of_month","_atf_month_copy" as "atf_month"

| fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"

 

anomaly_flakiness_runtime_criteria.png

And the code generated by the anomaly detection app -

 

``` Same data as above ```

| dedup _time
| sort 0 _time 
| table _time XXXX
| interpolatemissingvalues value_field="XXXX"
| fit AutoAnomalyDetection XXXX job_name=test sensitivity=1
| table _time, XXXX, isOutlier, anomConf

 

 

The major code difference is that with MLTK, we use -

 

| fit DensityFunction "median(total_run_time_mins)" by "atf_hour_of_day" dist=expon threshold=0.01 show_density=true show_options="feature_variables,split_by,params" into "_exp_draft_ca4283816029483bb0ebe68319e5c3e7"

 

whereas with the anomaly detection app, we use -

 

| fit AutoAnomalyDetection XXXX job_name=test sensitivity=1
| table _time, XXXX, isOutlier, anomConf

 

 

Any ideas why the fit function uses DensityFunction vs AutoAnomalyDetection parameters, and why the results are different?

Labels (1)
Tags (3)
0 Karma

ljvc
Explorer

DensityFunction and AutoAnomalyDetection are vastly different algorithms, so different results are to be expected. See Developing the Splunk App for Anomaly Detection | Splunk for more info on the Anomaly Detection App's custom algorithm and Algorithms in the Machine Learning Toolkit - Splunk Documentation for the MLTK's DensityFunction.

At least in my testing, the ADESCA/Earthgecko-Skyline stack in the Anomaly Detection App is more prone to alerting on non-cyclical low values when compared to the boundaries generated by the DensityFunction, though I have no good explanation for this behavior as of right now. 

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...