How to troubleshoot why hot buckets are not rolling after exceeding maxHotSpanSecs?

[craigwilkinson]

Hi All,

My hot bucket is not rolling when its span has exceeded maxhotspansecs. Could you please provide assistance?

We are currently using a Splunk index, purely for data archiving purposes with the requirements as per below:
- The data will be captured in single bucket of 24hour period for Reingestion purposes.
- The hot bucket will roll straight from Hot to Cold.
- Data will sit in cold for 6 days
- Data will roll to frozen after a period of 7 days.

After applying the configuration (indexes.conf as per below): I have noticed that the bucket span has exceeded 86401 as defined.
Bucket Start epoch time: 1481822441
Bucket End Epoch time: 1482106850

Hence Span sec = 284409 - which is greater than 86401.

Indexes.conf Snippet:

[my_index]
frozenTimePeriodInSecs = 604800     
maxTotalDataSizeMB = 400000             
maxWarmDBCount = 0                      
maxHotSpanSecs = 86401                  
maxHotBuckets = 1                               
coldToFrozenDir = 

Kind regards,

Craig

[esix_splunk · Accepted solution]

There is no rolling straight from Hot to Cold. So I would wonder why your approach has that in consideration. Why are you not letting hot buckets roll after 1 day into warm, and then warm to frozen after 7 days?

In regards to maxhotspansecs, this is a bounds and wont guarantee your buckets being aged out at exactly 1 day. There really isnt a good way to do this except to manually force a hot to warm roll at a set time everyday.

Here's a good link that addresses this also : https://answers.splunk.com/answers/2337/how-do-i-configure-my-indexes-so-that-hot-buckets-to-roll-to....

[damode]

Hi @esix [Splunk],

The above link is broken. Can you please share the updated link ?

Thanks,
Dev

[craigwilkinson]

Oh ok! This is news to me.

The indexes.conf suggests that it's possible to roll from Hot > Cold directly:
maxWarmDBCount =
* The maximum number of warm buckets.
* Warm buckets are located in the for the index.
* If set to zero, Splunk will not retain any warm buckets
(will roll them to cold as soon as it can)
* Highest legal value is 4294967295
* Defaults to 300.

Are you able to elaborate further as to why it's not achievable to roll directly from Hot > Cold ?
Or is this just known functionality?

[esix_splunk]

So technically, you can avoid having warm buckets yes. However, the pipelines for the hot -> warm -> cold -> frozen are not mutually exclusive. Meaning you can't skip the warm or cold buckets. You are just lowering the amount of time data is allowed to stay in these to a minimum. What this will reflect is i/o associated to a hot/warm roll, then (near) immediate roll to cold.

Again, whats your use case where you want to keep these out of warm? Whats your reason for this? What you have described above is very doable with a hot(1day) to warm(1day to 6days) to frozen(7days) roll.

Typically the only reason to roll from warm to cold is to age out data, or to move data to second tier storage, e.g., from ssd to spindles.

[craigwilkinson]

Ok, sure.

So if we increase maxWarmDBCount > 0, to say: maxWarmDBCount = 6.
This should fix the issue of Hot buckets not rolling when the hot bucket timespan exceeds: maxHotSpanSecs = 86401 ?

So our new configuration would be:

Indexes.conf Snippet:
[my_index]
frozenTimePeriodInSecs = 604800
maxTotalDataSizeMB = 400000
maxWarmDBCount = 0
maxHotSpanSecs = 86401
maxHotBuckets = 1
maxWarmDBCount = 6
coldToFrozenDir =

Will this work as expected?

Currently hot, warm and cold directories are on the same type of storage - however initially, there was a requirement to move to cold directory, as this would be cheaper disk space. As this is not the case anymore we can neglect this requirement.

[damode]

Hi @craigwilkinson ,

just curious to know, on what factor did you choose 6 for maxWarmDBcount ? is it the number of days ?
because I am facing the same issue. I had set maxHotSpanSecs = 2592000 [hot bucket - 30 days] and still hot bucket didnt roll to cold. I am not sure what maxWarmDBcount in this case.

[craigwilkinson]

Hey @damode,

Apologies, I think there was an error with my initial configuration.

We're currently running maxWarmDBCount=3.

I'm not 100% sure what the reasoning was as this was a year ago sorry.
But setting this value above 1 addressed the issue of buckets not rolling.

-Craig

[damode]

Thanks, @craigwilkinson.

[pmalcakdoj]

When maxHotBuckets=1, maxHotSpanSecs is ignored.

NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all events to the single hot bucket and maxHotSpanSeconds will not be enforced.

Because of this, hot bucket will now only be rolled due to size (ie. 400000MB in your case)