Splunk Search

Detecting outlier without using Splunk Machine Learning Toolkit

quahfamili
Path Finder

Hi all,

Not sure if it is a good way to solve this. Currently I do not have access to Splunk Machine Learning Toolkit due to computer management policies. I would like to check if I can use any self written logic in Splunk to detecting outliers with a given set of data.

Might be too broad of a question.

Thanks in advance.

Tags (1)
0 Karma

bowesmana
SplunkTrust
SplunkTrust

You can detect outliers in a number of ways, e.g.

| timechart span=1h count
| streamstats window=24 avg(count) as avg stdev(count) as std 
| eval m=2, lower=avg-(std * m), upper=avg+(std * m), outlier = if(count < lower OR count > upper, 1, 0)
| table _time upper lower count outlier

and then if you plot the outlier field on a chart overlay with a max Y axis of say, 2, you will have a vertical line showing the outliers

Adjust m to get a suitable standard deviation multiplier

niketn
Legend

@quahfamili, Can you install Splunk Machine Learning Toolkit (MLTK) on your personal machine? MLTK allows you to view underlying queries and macros for finding outliers, which you can then implement in your Splunk instance. However, if the query relies on an algorithm they can not be usable unless you install MLTK on the instance where you want to identify outlier.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

quahfamili
Path Finder

currently not possible to install on that system.

I was just thinking aloud to see if it is possible to write some simple algorithm in splunk, something like manual trend line with some threshold to build a model and check each data point against the model.

I actually hope I can install the MLTK to test but I cannot.

0 Karma

niketn
Legend

What I meant was for Standard Deviation, Interqartile Range and Mean Absolute deviation, you can use built in Splunk queries which are also used in Machine Learning Toolkit. These do not require MLTK to run but, in order to get those queries you will have to install MLTK somewhere (may be your personal laptop with Splunk Enterprise), not on the machine where you are building the Outlier detection.

If you want to create your own custom search command you can your Python SDK for Splunk. However, it is easier and better to integrate Machine Learning Algorithms via MLTK rather that starting from scratch on your own:

http://dev.splunk.com/view/python-sdk/SP-CAAAEU2
http://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutcustomsearchcommands

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

quahfamili
Path Finder

@niketnilay I do not have access to the splunk system side (no admin) too. Thanks for the reply I will read up on what you share. Might have to see if it is possible to python some logic in.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...