Collapsing multiple events

[Runals]

I'd like to collapse multiple firewall logs into very few events to help people understand connectivity between endpoints at a glance. The idea would be to show, over a given period of time, the first and last time of activity (action field) by source and destination IP. That is easy enough with a search like

... | stats min(_time) as start max(_time) as end by src_ip dest_ip action

However, what if traffic was being allowed, then blocked, then allowed again? You'd end up with just 2 events where the blocked time range would be included in the time range of the other event showing allowed traffic. What I'm looking for is 3 events where traffic between X and Y was allowed, then the time range where traffic was blocked, and then the timerange between the block being removed and the end of the query time selection.

I'm thinking streamstats is likely part of the answer, but I haven't played with it enough.

[Richfez]

I've been playing with the streamstats way and so far have had difficulty. I will continue to fool around with it.

One possible way (non-streamstats) would be

... | transaction src_ip dest_ip action 
| stats first(_time) AS begin, last(_time) AS end, dc(action) as Actions BY src_ip dest_ip action

You'll notice I left in dc(action) as Actions to double-check with. Looking through my example on my own network over the past few minutes I found all were "1" action (so no duplicates) and I found at least one item (3rd and 4th in my list) for one src_ip, dest_ip pair that had both an allowed line and a blocked line.

I think (but will have to think on it more and do some more testing to confirm) that stats doesn't respect time boundaries and will aggregate out any splits by time or whatever, whereas transaction won't. But if this will do you for now, maybe we can optimize/fix it later.

For reference in case you want to try the streamstats way, here's a couple of posts about things like this:

Counts for successive events but not across intervening events
Something else similar but in more detail.

My next stop was probably going to be re-examining the March 2016 Virtual .conf session (Nick Mealy, Sideview) on stats aggregation specifically to look for ways to erase the transaction and convert it to a stats directly. Or failing that, comparing those streamstats answers (that I think all contain transaction) to see if that may be a solution.

[Richfez]

Thinking this through some more, I don't think it's what you want.

Is this stream data or something like firewall logs?

I'm really not trying to be difficult, I think there's a bit more complexity in this question than I originally thought once I really started digging into a solution for it and digging into ways to do it "right". And that led me down the path to find out more precisely what it is that's desired.

Do you want specifically to close "transactions" every time action changes? So that if between two IPs over some time frame you had "allowed, allowed, blocked, allowed, allowed, blocked, blocked, allowed" would that be 5 "sessions"?

The problem with that is that I think they may overlap. Suppose we have...
1.1.1.2 -> 1.1.1.3 port 22 allowed (repeat 5 times)
1.1.1.2 -> 1.1.1.3 port 80 blocked (repeat 3 times)
1.1.1.2 -> 1.1.1.3 port 22 allowed (repeat 6 times, contiguous with the 5 events above).
Now, depending on various factors, this could be one ongoing ssh session with a blocked http attempt in the middle. If you don't see the ports involved, you'd have 5 allowed, 3 blocked, 6 allowed.

Would you want to see

src=1.1.1.2, dest=1.1.1.3, allowed_ports=22, allowed_ports_count=11, blocked_ports=80, blocked_ports_count=3

(in table format, though, but indenting and tabs are hard sometimes in this editor). This would possibly aggregate out the detail you may be wanting, but would be unambiguous in the face of changing actions across one ip->ip set of communication.

Or a set of three sessions for the three "groups",

1.1.1.2 -> 1.1.1.3 allowed, start1, end1
1.1.1.2 -> 1.1.1.3 blocked, start2, end2
1.1.1.2 -> 1.1.1.3 allowed, start3, end3

Where end1 may potentially equal start3 in this example. What if the ending of stream 1 above and the start of stream 2 overlap (e.g. sequence is start1, start2, end1, end2, start3, end3 <- should that period between start2 and end1 be considered allowed or blocked?)

Or is this always only on an individual packet level, where each packet, regardless of some sort of stream it's in will be considered its own and only grouped in sequence? So if the three blocked packets "interrupt" the steady ssh traffic, should that be considered a break in the allowed sequence then? That way if they overlap, it won't matter it'll just break it into more of this "version of a session?"

[Runals]

This sort of gets me close though the streamstats command 'remembers' the timestamp of the first src/dest IP and action. It appears there is a new argument in 6.4 which likely be part of the ultimate solution (reset_on_change) though because I'm including the time field it might balk at that. I'm not sure when we are going to upgrade to 6.4 though - maybe 6.4.2. From a processing perspective it this does sort of suck that you have to pull back so much data, sort it, do the streamstats, etc. There is still the needed step of collapsing the events. As for formatting I'm thinking more like the second example (set of three sessions for the three 'groups') though likely have the number of events.

...  | table _time src_ip dest_ip dest_port action | sort 0 src_ip dest_ip dest_port _time | streamstats min(_time) as start max(_time) as end by src_ip dest_ip dest_port action | convert ctime(start) ctime(end) | table _time src_ip dest_ip dest_port action start end

[Richfez]

I know! I saw that reset_on_change too and got all excited! It'll make several of these types of things much easier!

Will continue to think on it, though.

Might this be a candidate for a summary index? If you got something "slow" figured out pretty well then just ran it on a 5 minute interval dumping it into a summary index, you could end up with a fast, easily searched summary in the place of the slow, evil transaction/streamstats/whatever based stuff.

[Runals]

Not sure about summary index as I think in the end you still run into the same issue.

This is VERY close, I think, to what I want only it doesn't have the count of events. It also takes a bit of liberty with the start of when the action happened as it borrows from the end time of the previous action state as I'm trying to overcome not having reset_on_change. Lots and lots of sorting. Will see how this holds up.

... | eval dest_port = coalesce(dest_port, "-") | table _time src_ip dest_ip dest_port action | sort 0 src_ip dest_ip dest_port _time | streamstats min(_time) as start max(_time) as end count by src_ip dest_ip dest_port action | sort 0 src_ip dest_ip dest_port -_time | streamstats current=f window=1 last(action) as lastAction by src_ip dest_ip dest_port  | sort 0 src_ip dest_ip dest_port _time| convert ctime(start) ctime(end) | table _time src_ip dest_ip dest_port action start end count lastAction | where action != lastAction OR isnull(lastAction) | streamstats current=f window=1 last(end) as fooStart by src_ip dest_ip dest_port | eval fooStart = coalesce(fooStart,start) | table src_ip dest_ip dest_port fooStart end action | rename fooStart as start