Splunk Search

Scheduled search for populating summary index ignores the last 30 seconds of events

Sharzi
Explorer

Hello,

I recently faced an issue when populating a summary index. I scheduled a saved search to run every hour (with the last 60 minutes time range) and populate a summary index. The search takes around 5 minutes every time to be completed. My problem is that every time this scheduled search runs to populate the index, events in the last 30 seconds of the time range will be discarded from the results by Splunk. For example, for a one-hour time range like 9:00:00 to 10:00:00, the index is only populated with the events from 9:00:00 to 9:59:30. This issue caused some gaps and discrepancies in our index data.  Is there any way to solve this?

I searched a lot but couldn't find any answer 😞

Thanks.

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk.  That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

ITWhisperer
Legend

I usually schedule summary index updates at least 5 minutes past the hour for the previous hour, just to give the indexers time to do their work.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk.  That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

Sharzi
Explorer

Hi @richgalloway ,

We don't have any transaction command in our search query, but as you said, the problem was "latest".

I changed the "latest" and now it is working fine! Thanks.

 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!