Splunk performs tsidx reduction immediately on warm buckets

[jmangs]

I've recently upgraded to Splunk 6.6.0 and now seem to be having a problem with one of my indexes; every time I searched it, it would give a warning about reduced buckets that you normally see after tsidx reduction has been performed. Checking the latest warm buckets shows that the *.tsdx files have been replaced by *.mini.tsdx files.

I have tsidx reduction enabled, but it should only be reducing them after 30 days, not immediately. In addition, only this index in question has this problem. It's a relatively large index with 416.41 GB out of 488.28 GB in use. This wasn't an issue before on 6.5.3. I've tried settings auto_high_volume for the index but it seems to have no effect as of now.

Checking dbinspect I can see it setting warm buckets to mini immediately:

id  tsidxState  state   avg(sizeOnDiskMB)
4163    mini    warm    314.828125
4164    mini    warm    13.57421875
4165    full    hot 791.87109375
4166    full    hot 0.43359375
4703    mini    warm    307.6640625
4704    mini    warm    134.53125
4705    full    hot 760.03125
4706    full    hot 0.5078125
4707    full    hot 0.03125

After more investigation this seems to be related to logs that are received in the future due to a missing timezone; logs in UTC are received 4 hours in advance which then triggers an automatic tsidx reduction to occur automatically. Seems like a Splunk 6.6.0 bug.....

[mhoogcarspel_sp]

I reproduced this and reported this issue to Engineering and the fix is now in Splunk 6.6.2+

http://docs.splunk.com/Documentation/Splunk/6.6.2/ReleaseNotes/6.6.2
2017-06-08 SPL-142006, SPL-142492 TSIDX Reduction kicks in before newest event is old enough when events come in with future timestamp in Splunk 6.6.0

[ddrillic]

Gorgeous ; - )

[Justin_Brown]

We've been having the same problem. We've tested several scenarios and in every case, regardless of the event time stamp (future, past, present) if TSIDX reduction is enabled on the index it immediately goes into effect when a bucket migrates from hot to warm. This applies to 6.6.0 and 6.6.1. We also experimented with setting a timezone in the past just to see if it made a difference and it didn't. So it would seem if you're using this feature with on-prem Splunk 6.6.0 or newer you're probably wondering why searches seem significantly slower than prior to the update.

[jmangs]

Follow up to this, I think it's a bug in Splunk 6.6.0 - this actually occurs on any index which has tsidx reduction enabled and receives logs from the future sadly.

Haven't found a workaround yet aside from disabling the feature.