Getting Data In

Why did Splunk suddenly start indexing more than 100,000 Events on the same timestamp?

Path Finder

I have a single search that stores many events (~500,000) on the same timestamp.
As I understood, splunk chunks the data to 100,000 and stores the next 100,000 to the next second.
This works fine but, last week splunk stores all the 500,000 to the same timestamp so I can't read the data because I get

"Events may not be returned in sub-second order due to search memory limits configured in limits.conf:[search]:max_rawsize_perchunk. See search.log for more information."

and the search slows down massively.

Does anyone knows why splunk doesn't chunk the results as usual?

Best regards

0 Karma

Esteemed Legend

Yes, if you have a breakdown in your timestamping, Splunk will default to setting the untimestampable event to the timestamp of the previous event. In this case, you should see splunkd.log logs in index=_internal from your indexers like this:

DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event ...

The solution is to fix your broken timestamp configuration.

0 Karma
Get Updates on the Splunk Community!

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...

Adoption of Infrastructure Monitoring at Splunk

  Splunk's Growth Engineering team showcases one of their first Splunk product adoption-Splunk Infrastructure ...