topic Re: Scheduled search for populating summary index ignores the last 30 seconds of events in Splunk Search

Scheduled search for populating summary index ignores the last 30 seconds of events

Sharzi — Fri, 12 Mar 2021 18:50:41 GMT

Hello,

I recently faced an issue when populating a summary index. I scheduled a saved search to run every hour (with the last 60 minutes time range) and populate a summary index. The search takes around 5 minutes every time to be completed. My problem is that every time this scheduled search runs to populate the index, events in the last 30 seconds of the time range will be discarded from the results by Splunk. For example, for a one-hour time range like 9:00:00 to 10:00:00, the index is only populated with the events from 9:00:00 to 9:59:30. This issue caused some gaps and discrepancies in our index data. Is there any way to solve this?

I searched a lot but couldn't find any answer 😞

Thanks.

Re: Scheduled search for populating summary index ignores the last 30 seconds of events

richgalloway — Fri, 12 Mar 2021 20:30:36 GMT

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk. That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

Re: Scheduled search for populating summary index ignores the last 30 seconds of events

ITWhisperer — Fri, 12 Mar 2021 22:36:00 GMT

I usually schedule summary index updates at least 5 minutes past the hour for the previous hour, just to give the indexers time to do their work.

Re: Scheduled search for populating summary index ignores the last 30 seconds of events

Sharzi — Mon, 15 Mar 2021 15:47:44 GMT

Hi @richgalloway ,

We don't have any transaction command in our search query, but as you said, the problem was "latest".

I changed the "latest" and now it is working fine! Thanks.