All Apps and Add-ons

Anomalies detection using Splunk vs R

manishvrmv
New Member

Hi all,

I have a small numerical dataset to perform anomalies detection. My data contains 177 events and I have imputed 3 records to check anomalies using Splunk and R. I applied LOF algo. in R and find out all 3 events. In splunk, I applied anomalies command and generated unexpectedness score for each event and sorted in decreasing order. I found normal events as anomalous and imputed event comes into 6 place. Even all event found as with very less unexpectedness score.

I am just curious to know whether Splunk perform well with machine learning commands such as anomalies, outliers, cluster etc. one more query i have whether splunk work well with numerical data which contains timpstamp.

Tags (1)
0 Karma

prelert
Path Finder

If the dataset is something like:

_time, value
10:00:01, 3.22
10:00:04, 32.22
...

An effective approach to identify anomalies is to create a statistical model of the numerical values, and computing the probability of a specific data value. If the probability is low, then the value is anomalous.

Generally, to accurately model these data and avoid false positives these models needs to be more sophisticated than a simple Normal distribution. In addition, these data are generally periodic and so the models need to allow for daily and weekly patterns.

LOF methods can be effective on static low dimensional datasets, but suffer from similar issues to kernel density functions (overfitting, linear space complexity etc.).

Further details are available here:
http://www.ijmlc.org/papers/398-LC018.pdf

We have built an app to automatically identify anomalies in numeric and categorical data using these techniques:

http://apps.splunk.com/app/1306/

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...