Scheduled search for populating summary index ignores the last 30 seconds of events

[Sharzi]

Hello,

I recently faced an issue when populating a summary index. I scheduled a saved search to run every hour (with the last 60 minutes time range) and populate a summary index. The search takes around 5 minutes every time to be completed. My problem is that every time this scheduled search runs to populate the index, events in the last 30 seconds of the time range will be discarded from the results by Splunk. For example, for a one-hour time range like 9:00:00 to 10:00:00, the index is only populated with the events from 9:00:00 to 9:59:30. This issue caused some gaps and discrepancies in our index data.  Is there any way to solve this?

I searched a lot but couldn't find any answer 😞

Thanks.

[ITWhisperer]

I usually schedule summary index updates at least 5 minutes past the hour for the previous hour, just to give the indexers time to do their work.

[richgalloway · Accepted solution]

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk.  That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

[Sharzi]

Hi @richgalloway ,

We don't have any transaction command in our search query, but as you said, the problem was "latest".

I changed the "latest" and now it is working fine! Thanks.