topic Stdev against Null Time Bucket- How to create a rule alert on internal IP scanning? in Deployment Architecture

Stdev against Null Time Bucket- How to create a rule alert on internal IP scanning?

ericl42 — Mon, 16 May 2022 15:17:07 GMT

I've read quite a few forum posts about this but honestly didn't find a great solution for my use case (note that is probably because I didn't fully understand some of the items going on).

I'm trying to create a rule to alert on internal IP scanning. Below is my current logic.

index="firewall*"
| bucket _time span=15m@m
| stats dc(dest_ip) as num_dest_ips values(dest_ip) as dest_ips dc(dest_port) as num_dest_ports values(dest_port) as dest_ports count by src_ip, _time
| eventstats avg(num_dest_ips) as avg stdev(num_dest_ips) as stdev by src_ip | eval avg = round(avg,2) | eval stdev = round(stdev,2)
| lookup dnslookup clientip AS src_ip OUTPUT clienthost as src_dns
| eval temp=split(src_dns,".") | eval src_dns=mvindex(temp,0)
| eval src_system = coalesce(src_dns, src_ip)
| search NOT src_system=dns*
| eval lower_bound = avg-(stdev*.3) | eval lower_bound = round(lower_bound,2)
| eval upper_bound = avg+(stdev*.3) | eval upper_bound = round(upper_bound,2)
| eval isOutlier = if(dest_ips>upper_bound OR dest_ips<lower_bound,1,0)
| eval difference = upper_bound-lower_bound | eval different = round(different,2)
| table _time, src_ip, src_system, dest_ips, num_dest_ips, dest_ports, num_dest_ports, count, avg, lower_bound, upper_bound, difference, stdev, isOutlier

This works very well but as I mentioned, if 192.168.1.1 has no entries during a 15 min time bucket, then no entries will show up and therefore the math won't occur.

I've seen some posts where they append data and make the bucket command work but this is a fairly large index and I'd prefer not to double up on the search. I've seen others that recommend using timechart but my confusion on that front is how I would get all of my various stats commands to work with it.

Overall, I Just want something like the table below to occur and the 9:15 line be populated to zero if there are truly no entries. Unless there is a better way to do this entire setup, which I'm definitely open to.

_time src_ip num_dest_ips avg
9:00 192.168.1.1 30 15
9:15 192.168.1.1 0 15

Any hep would be greatly appreciated.

Below is what I would like to see.

Re: Stdev against Null Time Bucket

to4kawa — Wed, 11 Dec 2019 12:09:35 GMT

 index="firewall*"
 | bucket _time span=15m@m
 | timechart limit=0 cont=f dc(dest_ip) as num_dest_ips values(dest_ip) as dest_ips dc(dest_port) as num_dest_ports values(dest_port) as dest_ports count by src_ip
 | eventstats avg(num_dest_ips) as avg stdev(num_dest_ips) as stdev by src_ip | eval avg = round(avg,2) | eval stdev = round(stdev,2)
 | lookup dnslookup clientip AS src_ip OUTPUT clienthost as src_dns
 | eval temp=split(src_dns,".") | eval src_dns=mvindex(temp,0)
 | eval src_system = coalesce(src_dns, src_ip)
 | search NOT src_system=dns*
 | eval lower_bound = avg-(stdev*.3) | eval lower_bound = round(lower_bound,2)
 | eval upper_bound = avg+(stdev*.3) | eval upper_bound = round(upper_bound,2)
 | eval isOutlier = if(dest_ips>upper_bound OR dest_ips<lower_bound,1,0)
 | eval difference = upper_bound-lower_bound | eval different = round(different,2)
 | table _time, src_ip, src_system, dest_ips, num_dest_ips, dest_ports, num_dest_ports, count, avg, lower_bound, upper_bound, difference, stdev, isOutlier

I changed it a little.

Re: Stdev against Null Time Bucket

ericl42 — Wed, 11 Dec 2019 14:30:13 GMT

Thanks for your response. I tried your query and the only thing I got was two rows that did a 9:00 and 9:15 bucket but no other data is in any of the other columns.

Attaching a screenshot to my initial question of what I would like to see and then I'll only alert where outLier>0.

Re: Stdev against Null Time Bucket

to4kawa — Wed, 11 Dec 2019 17:56:04 GMT

Standard_deviation

the time has only one ip, so stdev is 0.
It can't be helped.

  | eval isOutlier = if(stdev !=0 AND (dest_ips>upper_bound OR dest_ips<lower_bound),1,0)

How about changing the conditions?

Re: Stdev against Null Time Bucket

ericl42 — Wed, 30 Sep 2020 03:18:52 GMT

I like the concept of potentially changing the logic if I can't make the other time show num_dest_ips as 0. However, I could see there being issues where if two hosts have the same amount of IPs during both buckets (e.g. 20), then the stdev is going to be 0 there too.

Re: Stdev against Null Time Bucket

ericl42 — Thu, 12 Dec 2019 19:09:13 GMT

I finally found a solution using a combination of timechart which instantly adds 0 to empty bucks and then untabling it so I can then format it however I want.

| timechart span=5m limit=0 dc(dest_ip) as num_dest_ips by src_ip
| untable _time, src_ip, num_dest_ips
| eventstats avg(num_dest_ips) as avg stdev(num_dest_ips) as stdev by src_ip | eval avg = round(avg,2) | eval stdev = round(stdev,2)

Re: Stdev against Null Time Bucket

ErikA — Mon, 16 May 2022 14:18:20 GMT

Yes, this worked for me as well. Thanks!