topic Re: Is using count over streamstats time_window not timechart span possible? in Splunk Search

Is using count over streamstats time_window not timechart span possible?

RayLio — Thu, 05 Jan 2017 19:38:55 GMT

Hello splunkfans,

i'm kind of running out of ideas and this is my first contact to streamstats. 😕
I am working on a statistic of botnet portscans on my firewalllogs. The goal is based on the firewallevents on a specific interface and direction, to show how many different public IPs have scanned how many dest_ports, events=attacks, and in what timerange.
The break where i assume it is a botnet portscan is when i get over 150 events in a 2 min flow.

Here is my search:

index="security" interface=igb0  direction=in | streamstats time_window=2min count(_raw) as "attacks" distinct_count(source_ip) as "attackers", distinct_count(destination_port) as "attacked ports"  | search attacks>150

What i am looking at due to my search is a nice timeline over the last 3 days with 3 peaks that represent the time_window=2min of my streamstats.
My goal is just to get a table of these peaks and the timerange they occured.
Like this: table attacks, attackers, "attacked ports", //"timerange first and last event"//

Problem i have is that this table without time (as i have no solution for that), shows me all streamstats events like this:

attacks     attackers       attacked ports  
151            34             9
152            34             9
153            34             9
154            34             10
155            34             10
156            34             10
157            34             10

Searchresult of just the streamstats is of cause the events themselfs that are relevant in the time_window.

How can i get just the peaks of this and the timerange between the first and last event of these?
I tried so many combos with max() and top, but as i dont know how many peaks will occure, i cant regulate the top.
A timechart seams to be a dirty solution with a span=2min but a portscan can happen between 00:01:45 and 00:02:15 and would be split in half and not recognised.

Anyone can guide me in the right direction on how to get the tops of this stream? 🙂

Re: Is using count over streamstats time_window not timechart span possible?

niketn — Tue, 29 Sep 2020 12:18:17 GMT

Try using first() and last() statistical functions:

index="security" interface=igb0  direction=in source_ip=* destination_port=*
| streamstats time_window=2m count as "attacks" dc(source_ip) as "attackers", dc(destination_port) as "attacked ports" first(_time) as FirstTime last(_time) as LastTime 
| search attacks>150 
| eval FirstTime=strftime(FirstTime,"%c")
| eval LastTime=strftime(LastTime,"%c")
| table attacks attackers "attacked ports" FirstTime LastTime

PS: I have added source_ip=* and destination_port=* assuming they are always present. If not remove that from the base search. I have used strftime to convert to String Date Format.

Re: Is using count over streamstats time_window not timechart span possible?

DalJeanis — Thu, 05 Jan 2017 21:28:22 GMT

If you are looking for just the peaks, and not the individual events, then you probably want to use timechart instead.

index="security" interface=igb0  direction=in 
| timechart span=2min count as "attacks", dc(source_ip) as "attackers", dc(destination_port) as "attacked ports", min(_time) as "start Time", max(_time) as "End Time"  | search attacks>150

fix formatting

Re: Is using count over streamstats time_window not timechart span possible?

RayLio — Thu, 05 Jan 2017 23:14:52 GMT

We are getting close, thanks for the replies!

I tried this like niketnilay answered with modifications. It is very close, just have to get rid of the overlaps, see below.

index="security" interface=igb0  direction=in
| streamstats time_window=2min count as "attacks" dc(source_ip) as "attackers", dc(destination_port) as "attacked ports" first(_time) as FirstTime last(_time) as LastTime
| search attacks>150 
| eval FirstTime=strftime(FirstTime,"%c") 
| eval LastTime=strftime(LastTime,"%c") 
| table attacks attackers "attacked ports" FirstTime LastTime 
| sort -attacks 
| dedup FirstTime

Got me this:

attacks attackers attacked ports FirstTime LastTime

902 90 106 Tue Jan 3 16:34:06 2017 Tue Jan 3 16:32:12 2017
693 69 81 Tue Jan 3 00:15:08 2017 Tue Jan 3 00:13:19 2017
691 68 79 Tue Jan 3 00:14:26 2017 Tue Jan 3 00:12:30 2017
565 91 22 Tue Jan 3 16:34:31 2017 Tue Jan 3 16:32:34 2017
426 70 14 Tue Jan 3 00:15:41 2017 Tue Jan 3 00:13:42 2017
371 70 13 Tue Jan 3 00:15:46 2017 Tue Jan 3 00:13:47 2017
339 34 44 Wed Jan 4 23:23:57 2017 Wed Jan 4 23:22:16 2017
264 33 41 Wed Jan 4 23:23:03 2017 Wed Jan 4 23:21:04 2017
262 91 19 Tue Jan 3 16:34:54 2017 Tue Jan 3 16:32:55 2017
248 34 41 Wed Jan 4 23:22:59 2017 Wed Jan 4 23:21:01 2017
239 35 12 Wed Jan 4 23:24:32 2017 Wed Jan 4 23:22:33 2017
202 35 41 Wed Jan 4 23:22:50 2017 Wed Jan 4 23:20:51 2017

Yes DalJeanis i'm just looking for the peaks, but a timechart isn't accurate if the attack is cut into two time spans. Am i wrong?

I'm wondering why these dont have the same firsttime... There are overlaps in the timeranges. This is due to the search attacks>150 after the streamline.
Any chance to get this into the stream as a condition?