I have this issue on our indexes- it seems frozenTimePeriodInSecs and maxHotSpanSecs is not working
Buckets are over the frozenTimePeriodInSecs but still on hot buckets.
[skype] coldPath = volume:cold/skype/colddb coldPath.maxDataSizeMB = 400000 coldToFrozenDir = $SPLUNK_HOME/frozen/skype frozenTimePeriodInSecs = 8035200 homePath = volume:primary/skype/db homePath.maxDataSizeMB = 400000 maxDataSize = auto_high_volume maxHotBuckets = 10 maxHotSpanSecs = 7776000 maxTotalDataSizeMB = 400000
Thanks in advance.
It looks like you have timestamp issue or you are ingesting very old logs into Splunk and due to that Splunk is creating multiple hot buckets.
And based on my knowledge those buckets will roll from Hot to Warm when
maxHotSpanSecs whichever hit first and when you have more than 10 hot buckets based on your configuration (which means when 11th hot bucket will create, oldest hot bucket will roll from hot to warm), if you do not hit 10 hot buckets in that case bucket will only roll when splunk will restart or
maxHotSpanSecs whichever hit first.
In your case it looks like bucket with ID 134 created but didn't hit
maxHotSpanSecs and this bucket will only roll when you will restart splunk or more data will be ingested in this bucket & when it reaches
maxHotSpanSecs, if you do not restart splunk or bucket will not reach
maxHotSpanSecs in that case it will sit as Idle hot bucket and by default
maxHotIdleSecs setting is 0 which means infinite time (A value of 0 turns off the idle check). In this case either you need to fix timestamp issue if you have timestamp recognition problem on splunk or if timestamp recognition is correct but data is very old then you can set
maxHotIdleSecs to few days (For example : 7 or 14 days) and after this days if hot bucket will not receive any events then it will roll from hot to warm. Once this bucket with ID 134 will convert from hot to warm it will immediately remove because it already reached
Thanks for this:
I do restart he indexers and do a rolling restart and after that, some of the buckets are rolled and others are being frozen.
Yes, we are still facing a timespan issue due to logs are hard to understand.