topic Re: Detecting outliers query in Splunk Search

Detecting outliers query

muradghazzawi — Sun, 03 May 2020 21:54:57 GMT

Hi everyone
I was reading through "endpoint security analyst with Splunk (online experience)" which you can find here:
http://si_usecase_02.splunkoxygen.com/en-US/app/OLE_Security_Endpoint/sec_search_01?tour=gs_main_intro

This is a four exercises tutorial that will show you how to detect and prevent advanced malware,
anyway I was moving along with the tutorial step by step and this statement caught my attention:

Any process activities with a command line command length that is more than four times the average and standard deviation command line command lengths for each host is an outlier

My question is why?
is this a standard formula?
an axiom?
where did this come from?

Here is the query that was used in this tutorial:

sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
| eval cmdlen=len(CommandLine)
| eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
| stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
| eval threshold = 4 * ( stdevperhost + avgperhost )
| where maxlen > threshold

Here you can see it in the second eval command (in the line before the last).

Thanks in advance.

Re: Detecting outliers query

to4kawa — Sun, 03 May 2020 23:26:50 GMT

My question is why?: These are the techniques and best practices how to detect and prevent ransomware infections.
is this a standard formula?: no
an axiom?: no
where did this come from? best practices

https://www.splunk.com/en_us/form/security-investigation-online-experience-endpoint.html

Once ransomware hits, you have a very short window of time to detect it and respond, before critical business data is encrypted and therefore no longer accessible to you, your business, or your customers. This sandbox environment guides you, in a safe environment with "live threats," through techniques and best practices on how to detect and prevent ransomware infections.

How about this as an example of a long string of commands?

http://unit42.elegance.work/unit42-pulling-back-the-curtains-on-encodedcommand-powershell-attacks/

Re: Detecting outliers query

tauliang — Sun, 03 May 2020 23:39:18 GMT

This doesn't make much sense, and the threshold is too relaxed for anomaly detection. Statistically speaking, if it is 3-sigma or 4-sigma away from the mean/average, it is considered an outlier/anomaly. I think it was a typo in the tutorial, and it should be like this:

 sourcetype=xmlwineventlog:microsoft-windows-sysmon/operational EventCode=1
 | eval cmdlen=len(CommandLine)
 | eventstats avg(cmdlen) as avg, stdev(cmdlen) as stdev by host
 | stats max(cmdlen) as maxlen, values(avg) as avgperhost, values(stdev) as stdevperhost by host, CommandLine
 | eval threshold = avgperhost + 4 *stdevperhost 
 | where maxlen > threshold

Note I changed the the 5th line of the search to | eval threshold = avgperhost + 4 *stdevperhost .

Re: Detecting outliers query

tauliang — Sun, 03 May 2020 23:41:14 GMT

OK. I found something published on Splunk website, pay attention to 5m40s mark of the video tutorial.

Re: Detecting outliers query

to4kawa — Sun, 03 May 2020 23:52:12 GMT

4 *stdevperhost → 3 * stdevperhost (99.7%)
I think this is good enough for me.

Re: Detecting outliers query

muradghazzawi — Mon, 04 May 2020 11:46:26 GMT

Thank you, both of you @tauliang @to4kawa

Re: Detecting outliers query

tauliang — Mon, 04 May 2020 15:58:13 GMT

@muralikoppula you are most welcome. If it answers your question, could you please accept it as an answer? Thanks.