I've recently upgraded to Splunk 6.6.0 and now seem to be having a problem with one of my indexes; every time I searched it, it would give a warning about reduced buckets that you normally see after tsidx reduction has been performed. Checking the latest warm buckets shows that the *.tsdx files have been replaced by *.mini.tsdx files.
I have tsidx reduction enabled, but it should only be reducing them after 30 days, not immediately. In addition, only this index in question has this problem. It's a relatively large index with
416.41 GB out of
488.28 GB in use. This wasn't an issue before on
6.5.3. I've tried settings
auto_high_volume for the index but it seems to have no effect as of now.
dbinspect I can see it setting warm buckets to mini immediately:
id tsidxState state avg(sizeOnDiskMB) 4163 mini warm 314.828125 4164 mini warm 13.57421875 4165 full hot 791.87109375 4166 full hot 0.43359375 4703 mini warm 307.6640625 4704 mini warm 134.53125 4705 full hot 760.03125 4706 full hot 0.5078125 4707 full hot 0.03125
After more investigation this seems to be related to logs that are received in the future due to a missing timezone; logs in UTC are received 4 hours in advance which then triggers an automatic tsidx reduction to occur automatically. Seems like a Splunk 6.6.0 bug.....
I reproduced this and reported this issue to Engineering and the fix is now in Splunk 6.6.2+
2017-06-08 SPL-142006, SPL-142492 TSIDX Reduction kicks in before newest event is old enough when events come in with future timestamp in Splunk 6.6.0
We've been having the same problem. We've tested several scenarios and in every case, regardless of the event time stamp (future, past, present) if TSIDX reduction is enabled on the index it immediately goes into effect when a bucket migrates from hot to warm. This applies to 6.6.0 and 6.6.1. We also experimented with setting a timezone in the past just to see if it made a difference and it didn't. So it would seem if you're using this feature with on-prem Splunk 6.6.0 or newer you're probably wondering why searches seem significantly slower than prior to the update.
Follow up to this, I think it's a bug in Splunk 6.6.0 - this actually occurs on any index which has
tsidx reduction enabled and receives logs from the future sadly.
Haven't found a workaround yet aside from disabling the feature.