Knowledge Management
Highlighted

Adding "late" events to a Summary Index

Path Finder

We have a Summary Index saved search that uses a 5 minute sliding window and runs every 5 minutes. Sometimes events arrive several minutes or hours late and the Summary Indexing window for the event has already passed, thus leaving the summary index inconsistent with the original index.

Our strategy at the moment relies on events arriving within 5 minutes, our search's parameters are:

 earliest="-10m@m" latest="-5m@m"

We keep this window small, as we need dashboards that report in near real-time.

Is there a good strategy to detect events not present in a summary index due to this issue? Or perhaps a way to tell when an event arrives with a big difference in the real indexed time vs. the timestamp as reported by the event?

Highlighted

Re: Adding "late" events to a Summary Index

Splunk Employee
Splunk Employee

Detecting late events in general is fairly straightforward. The values of _time and _indextime are unix time_t style integers of those two values.

search terms | where (_indextime - _time) > 300

would show you events that came over 5 minutes late, for example.

The summary index frequently doesn't have the original events, only aggregate information about them, so I can't give a strategy for detecting what might have been missed in the general case.

View solution in original post

Highlighted

Re: Adding "late" events to a Summary Index

Path Finder

This will definitely help get us there. Thank you very much.

0 Karma
Highlighted

Re: Adding "late" events to a Summary Index

Builder

You can use transaction to collect and throw away duplicates. This will need some editing for your environment.

( search_terms ) OR ( index=summary search_name=searchName ) | transaction _time, other_identifying_fields keepevicted=true | where eventcount=1 | sistats count by _time, other_identifying_fields
0 Karma