All Apps and Add-ons

Anomalies detection using Splunk vs R

manishvrmv
New Member

Hi all,

I have a small numerical dataset to perform anomalies detection. My data contains 177 events and I have imputed 3 records to check anomalies using Splunk and R. I applied LOF algo. in R and find out all 3 events. In splunk, I applied anomalies command and generated unexpectedness score for each event and sorted in decreasing order. I found normal events as anomalous and imputed event comes into 6 place. Even all event found as with very less unexpectedness score.

I am just curious to know whether Splunk perform well with machine learning commands such as anomalies, outliers, cluster etc. one more query i have whether splunk work well with numerical data which contains timpstamp.

Tags (1)
0 Karma

prelert
Path Finder

If the dataset is something like:

_time, value
10:00:01, 3.22
10:00:04, 32.22
...

An effective approach to identify anomalies is to create a statistical model of the numerical values, and computing the probability of a specific data value. If the probability is low, then the value is anomalous.

Generally, to accurately model these data and avoid false positives these models needs to be more sophisticated than a simple Normal distribution. In addition, these data are generally periodic and so the models need to allow for daily and weekly patterns.

LOF methods can be effective on static low dimensional datasets, but suffer from similar issues to kernel density functions (overfitting, linear space complexity etc.).

Further details are available here:
http://www.ijmlc.org/papers/398-LC018.pdf

We have built an app to automatically identify anomalies in numeric and categorical data using these techniques:

http://apps.splunk.com/app/1306/

0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...