Transaction or Stats - need multiple starts and ends without the hundreds of thousands of events between each start & end

[mrpaul]

I am looking for a way to identify the start and end of a burst of events that has hundreds of thousands of events in each burst. I don't need the individual events themselves. This sounds like a perfect use for stats, except, multiple bursts can happen during the search period, and I need the start and end times of each burst.

For example, I may run a search over the past 24 hours, and during that time, there may be three or four bursts.

I can assume if there is a gap of 5 minutes or more, the burst has ended. Any given burst may last for minutes or an hour or so.

I have tried using a transaction with maxpause=5m and maxopenevents=100000, but the huge number of events is causing problems.

The events are all similar with no identifiers I could use to filter out everything but the start and end events (that would have simplified this!)

Thanks for your thoughts!

Mr Paul

[woodcock · Accepted solution]

Let's assume you have a field called status that has (at least) values like start and end; then you can use streamstats like this:

... | streamstats count earliest(_time) AS startTime latest(_time) AS endTime latest(status) AS lastStatus | count(eval(status="start")) AS sessionID | stats values(*) AS * BY sessionID

[sideview]

Sounds like you need a different approach that uses eval, streamstats and stats. If you can identify the start and end of the bursts with transaction, then chances are you can paint some marker fields onto these events with eval.

Can you post specific examples of the start and end events as well as some of the 'filler' in between? I can update this comment and then turn it into an answer.