ideas for reliably bracketing timerange around discrete 'snapshots'

[sideview]

Say that you have a huge volume of events, and they come in big batches. Each batch is a discrete unit, and mixing information from the most recent batch with the previous batch is unacceptable.

more givens:

1. the events within a particular batch are spread out over a few minutes.
2. we do have control over the data so we could write a particular event at the start and at the end of the batch if necessary. We could even create a start/end event that had a different source or sourcetype.

Given all this, Is there a good clean way to construct a custom search or a custom view that will be sure to operate only on the events of the most recent batch?

[Stephen_Sorkin · Accepted solution]

Your best bet is to use the "head" command which can take a predicate instead of an absolute count.

For example, the following search only takes (all of) the events from the most recent second from index=_internal:

index=_internal | streamstats dc(_time) as dc_time | head dc_time==1

[Stephen_Sorkin]

This ability of head has been around since 4.1, I believe.

[sideview]

transaction wont work here because the set of events needs to be sliced and diced up a number of ways by a lot of different 'stats foo(bar) by baz, bat' searches, and transaction is going to put me in multivalue hell.

[Brian_Osburn]

Wouldn't this be a good use of a transaction command? especially if you've got a well defined start and stop?

[sideview]

The events here are not in a single second, but this offers a tool that seems to open up a number of other ideas. Is that ability of the head command new in 4.2? It seems like I could use eval and streamstats to keep track of when I see the 'start' event and 'end' event, and then use head to terminate once I get back to the correct head event. Is that what you would do?