Adding "late" events to a Summary Index

[stephanbuys]

We have a Summary Index saved search that uses a 5 minute sliding window and runs every 5 minutes. Sometimes events arrive several minutes or hours late and the Summary Indexing window for the event has already passed, thus leaving the summary index inconsistent with the original index.

Our strategy at the moment relies on events arriving within 5 minutes, our search's parameters are:

 earliest="-10m@m" latest="-5m@m"

We keep this window small, as we need dashboards that report in near real-time.

Is there a good strategy to detect events not present in a summary index due to this issue? Or perhaps a way to tell when an event arrives with a big difference in the real indexed time vs. the timestamp as reported by the event?

[BobM]

You can use transaction to collect and throw away duplicates. This will need some editing for your environment.

( search_terms ) OR ( index=summary search_name=searchName ) | transaction _time, other_identifying_fields keepevicted=true | where eventcount=1 | sistats count by _time, other_identifying_fields

[jrodman · Accepted solution]

Detecting late events in general is fairly straightforward. The values of _time and _indextime are unix time_t style integers of those two values.

search terms | where (_indextime - _time) > 300

would show you events that came over 5 minutes late, for example.

The summary index frequently doesn't have the original events, only aggregate information about them, so I can't give a strategy for detecting what might have been missed in the general case.

[stephanbuys]

This will definitely help get us there. Thank you very much.