Splunk Search

Scheduled search for populating summary index ignores the last 30 seconds of events

Sharzi
Explorer

Hello,

I recently faced an issue when populating a summary index. I scheduled a saved search to run every hour (with the last 60 minutes time range) and populate a summary index. The search takes around 5 minutes every time to be completed. My problem is that every time this scheduled search runs to populate the index, events in the last 30 seconds of the time range will be discarded from the results by Splunk. For example, for a one-hour time range like 9:00:00 to 10:00:00, the index is only populated with the events from 9:00:00 to 9:59:30. This issue caused some gaps and discrepancies in our index data.  Is there any way to solve this?

I searched a lot but couldn't find any answer 😞

Thanks.

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk.  That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

I usually schedule summary index updates at least 5 minutes past the hour for the previous hour, just to give the indexers time to do their work.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There's often (usually) a delay from when an event is generated and when it is searchable within Splunk.  That delay can exceed 30 seconds and is why many Enterprise Security searches use latest=-1m rather than latest=now.

There may be other explanations, like a transaction command discarding events because they aren't (yet) part of a complete transaction.

---
If this reply helps you, Karma would be appreciated.

Sharzi
Explorer

Hi @richgalloway ,

We don't have any transaction command in our search query, but as you said, the problem was "latest".

I changed the "latest" and now it is working fine! Thanks.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...