Splunk Search

Detecting outliers query

muradghazzawi
Engager

Hi everyone
I was reading through "endpoint security analyst with Splunk (online experience)" which you can find here:
http://si_usecase_02.splunkoxygen.com/en-US/app/OLE_Security_Endpoint/sec_search_01?tour=gs_main_int...

This is a four exercises tutorial that will show you how to detect and prevent advanced malware,
anyway I was moving along with the tutorial step by step and this statement caught my attention:

Any process activities with a command line command length that is more than four times the average and standard deviation command line command lengths for each host is an outlier

My question is why?
is this a standard formula?
an axiom?
where did this come from?

Here is the query that was used in this tutorial:

sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
| eval cmdlen=len(CommandLine)
| eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
| stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
| eval threshold = 4 * ( stdevperhost + avgperhost )
| where maxlen > threshold 

Here you can see it in the second eval command (in the line before the last).

Thanks in advance.

0 Karma
1 Solution

tauliang
Communicator

This doesn't make much sense, and the threshold is too relaxed for anomaly detection. Statistically speaking, if it is 3-sigma or 4-sigma away from the mean/average, it is considered an outlier/anomaly. I think it was a typo in the tutorial, and it should be like this:

 sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
 | eval cmdlen=len(CommandLine)
 | eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
 | stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
 | eval threshold = avgperhost + 4 *stdevperhost 
 | where maxlen > threshold 

Note I changed the the 5th line of the search to | eval threshold = avgperhost + 4 *stdevperhost .

View solution in original post

tauliang
Communicator

This doesn't make much sense, and the threshold is too relaxed for anomaly detection. Statistically speaking, if it is 3-sigma or 4-sigma away from the mean/average, it is considered an outlier/anomaly. I think it was a typo in the tutorial, and it should be like this:

 sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
 | eval cmdlen=len(CommandLine)
 | eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
 | stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
 | eval threshold = avgperhost + 4 *stdevperhost 
 | where maxlen > threshold 

Note I changed the the 5th line of the search to | eval threshold = avgperhost + 4 *stdevperhost .

tauliang
Communicator

OK. I found something published on Splunk website, pay attention to 5m40s mark of the video tutorial.

0 Karma

to4kawa
Ultra Champion

4 *stdevperhost3 * stdevperhost (99.7%)
I think this is good enough for me.

muradghazzawi
Engager

Thank you, both of you @tauliang @to4kawa

0 Karma

tauliang
Communicator

@muralikoppula you are most welcome. If it answers your question, could you please accept it as an answer? Thanks.

0 Karma

to4kawa
Ultra Champion

My question is why?: These are the techniques and best practices how to detect and prevent ransomware infections.
is this a standard formula?: no
an axiom?: no
where did this come from? best practices

https://www.splunk.com/en_us/form/security-investigation-online-experience-endpoint.html

Once ransomware hits, you have a very short window of time to detect it and respond, before critical business data is encrypted and therefore no longer accessible to you, your business, or your customers. This sandbox environment guides you, in a safe environment with "live threats," through techniques and best practices on how to detect and prevent ransomware infections.

How about this as an example of a long string of commands?

http://unit42.elegance.work/unit42-pulling-back-the-curtains-on-encodedcommand-powershell-attacks/

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...