Splunk Search
Highlighted

ideas for reliably bracketing timerange around discrete 'snapshots'

SplunkTrust
SplunkTrust

Say that you have a huge volume of events, and they come in big batches. Each batch is a discrete unit, and mixing information from the most recent batch with the previous batch is unacceptable.

more givens:

  1. the events within a particular batch are spread out over a few minutes.
  2. we do have control over the data so we could write a particular event at the start and at the end of the batch if necessary. We could even create a start/end event that had a different source or sourcetype.

Given all this, Is there a good clean way to construct a custom search or a custom view that will be sure to operate only on the events of the most recent batch?

Tags (1)
Highlighted

Re: ideas for reliably bracketing timerange around discrete 'snapshots'

Motivator

Your best bet is to use the "head" command which can take a predicate instead of an absolute count.

For example, the following search only takes (all of) the events from the most recent second from index=_internal:

index=_internal | streamstats dc(_time) as dc_time | head dc_time==1

View solution in original post

Highlighted

Re: ideas for reliably bracketing timerange around discrete 'snapshots'

SplunkTrust
SplunkTrust

The events here are not in a single second, but this offers a tool that seems to open up a number of other ideas. Is that ability of the head command new in 4.2? It seems like I could use eval and streamstats to keep track of when I see the 'start' event and 'end' event, and then use head to terminate once I get back to the correct head event. Is that what you would do?

0 Karma
Highlighted

Re: ideas for reliably bracketing timerange around discrete 'snapshots'

Builder

Wouldn't this be a good use of a transaction command? especially if you've got a well defined start and stop?

0 Karma
Highlighted

Re: ideas for reliably bracketing timerange around discrete 'snapshots'

SplunkTrust
SplunkTrust

transaction wont work here because the set of events needs to be sliced and diced up a number of ways by a lot of different 'stats foo(bar) by baz, bat' searches, and transaction is going to put me in multivalue hell.

0 Karma
Highlighted

Re: ideas for reliably bracketing timerange around discrete 'snapshots'

Motivator

This ability of head has been around since 4.1, I believe.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.