Detecting outliers query

[muradghazzawi]

Hi everyone
I was reading through "endpoint security analyst with Splunk (online experience)" which you can find here:
http://si_usecase_02.splunkoxygen.com/en-US/app/OLE_Security_Endpoint/sec_search_01?tour=gs_main_int...

This is a four exercises tutorial that will show you how to detect and prevent advanced malware,
anyway I was moving along with the tutorial step by step and this statement caught my attention:

Any process activities with a command line command length that is more than four times the average and standard deviation command line command lengths for each host is an outlier

My question is why?
is this a standard formula?
an axiom?
where did this come from?

Here is the query that was used in this tutorial:

sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
| eval cmdlen=len(CommandLine)
| eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
| stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
| eval threshold = 4 * ( stdevperhost + avgperhost )
| where maxlen > threshold 

Here you can see it in the second eval command (in the line before the last).

Thanks in advance.

[tauliang · Accepted solution]

This doesn't make much sense, and the threshold is too relaxed for anomaly detection. Statistically speaking, if it is 3-sigma or 4-sigma away from the mean/average, it is considered an outlier/anomaly. I think it was a typo in the tutorial, and it should be like this:

 sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
 | eval cmdlen=len(CommandLine)
 | eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
 | stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
 | eval threshold = avgperhost + 4 *stdevperhost 
 | where maxlen > threshold 

Note I changed the the 5th line of the search to | eval threshold = avgperhost + 4 *stdevperhost .

[tauliang]

OK. I found something published on Splunk website, pay attention to 5m40s mark of the video tutorial.

[to4kawa]

4 *stdevperhost → 3 * stdevperhost (99.7%)
I think this is good enough for me.

[muradghazzawi]

Thank you, both of you @tauliang @to4kawa

[tauliang]

@muralikoppula you are most welcome. If it answers your question, could you please accept it as an answer? Thanks.

[to4kawa]

My question is why?: These are the techniques and best practices how to detect and prevent ransomware infections.
is this a standard formula?: no
an axiom?: no
where did this come from? best practices

https://www.splunk.com/en_us/form/security-investigation-online-experience-endpoint.html

Once ransomware hits, you have a very short window of time to detect it and respond, before critical business data is encrypted and therefore no longer accessible to you, your business, or your customers. This sandbox environment guides you, in a safe environment with "live threats," through techniques and best practices on how to detect and prevent ransomware infections.

How about this as an example of a long string of commands?

http://unit42.elegance.work/unit42-pulling-back-the-curtains-on-encodedcommand-powershell-attacks/