Deployment Architecture

How to troubleshoot why hot buckets are not rolling after exceeding maxHotSpanSecs?

craigwilkinson
Path Finder

Hi All,

My hot bucket is not rolling when its span has exceeded maxhotspansecs. Could you please provide assistance?

We are currently using a Splunk index, purely for data archiving purposes with the requirements as per below:
- The data will be captured in single bucket of 24hour period for Reingestion purposes.
- The hot bucket will roll straight from Hot to Cold.
- Data will sit in cold for 6 days
- Data will roll to frozen after a period of 7 days.

After applying the configuration (indexes.conf as per below): I have noticed that the bucket span has exceeded 86401 as defined.
Bucket Start epoch time: 1481822441
Bucket End Epoch time: 1482106850

Hence Span sec = 284409 - which is greater than 86401.

Indexes.conf Snippet:

[my_index]
frozenTimePeriodInSecs = 604800     
maxTotalDataSizeMB = 400000             
maxWarmDBCount = 0                      
maxHotSpanSecs = 86401                  
maxHotBuckets = 1                               
coldToFrozenDir = 

Kind regards,

Craig

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

There is no rolling straight from Hot to Cold. So I would wonder why your approach has that in consideration. Why are you not letting hot buckets roll after 1 day into warm, and then warm to frozen after 7 days?

In regards to maxhotspansecs, this is a bounds and wont guarantee your buckets being aged out at exactly 1 day. There really isnt a good way to do this except to manually force a hot to warm roll at a set time everyday.

Here's a good link that addresses this also : https://answers.splunk.com/answers/2337/how-do-i-configure-my-indexes-so-that-hot-buckets-to-roll-to....

View solution in original post

esix_splunk
Splunk Employee
Splunk Employee

There is no rolling straight from Hot to Cold. So I would wonder why your approach has that in consideration. Why are you not letting hot buckets roll after 1 day into warm, and then warm to frozen after 7 days?

In regards to maxhotspansecs, this is a bounds and wont guarantee your buckets being aged out at exactly 1 day. There really isnt a good way to do this except to manually force a hot to warm roll at a set time everyday.

Here's a good link that addresses this also : https://answers.splunk.com/answers/2337/how-do-i-configure-my-indexes-so-that-hot-buckets-to-roll-to....

damode
Motivator

Hi @esix [Splunk],

The above link is broken. Can you please share the updated link ?

Thanks,
Dev

0 Karma

craigwilkinson
Path Finder

Oh ok! This is news to me.

The indexes.conf suggests that it's possible to roll from Hot > Cold directly:
maxWarmDBCount =
* The maximum number of warm buckets.
* Warm buckets are located in the for the index.
* If set to zero, Splunk will not retain any warm buckets
(will roll them to cold as soon as it can)
* Highest legal value is 4294967295
* Defaults to 300.

Are you able to elaborate further as to why it's not achievable to roll directly from Hot > Cold ?
Or is this just known functionality?

esix_splunk
Splunk Employee
Splunk Employee

So technically, you can avoid having warm buckets yes. However, the pipelines for the hot -> warm -> cold -> frozen are not mutually exclusive. Meaning you can't skip the warm or cold buckets. You are just lowering the amount of time data is allowed to stay in these to a minimum. What this will reflect is i/o associated to a hot/warm roll, then (near) immediate roll to cold.

Again, whats your use case where you want to keep these out of warm? Whats your reason for this? What you have described above is very doable with a hot(1day) to warm(1day to 6days) to frozen(7days) roll.

Typically the only reason to roll from warm to cold is to age out data, or to move data to second tier storage, e.g., from ssd to spindles.

0 Karma

craigwilkinson
Path Finder

Ok, sure.

So if we increase maxWarmDBCount > 0, to say: maxWarmDBCount = 6.
This should fix the issue of Hot buckets not rolling when the hot bucket timespan exceeds: maxHotSpanSecs = 86401 ?

So our new configuration would be:

Indexes.conf Snippet:
[my_index]
frozenTimePeriodInSecs = 604800
maxTotalDataSizeMB = 400000
maxWarmDBCount = 0
maxHotSpanSecs = 86401
maxHotBuckets = 1
maxWarmDBCount = 6
coldToFrozenDir =

Will this work as expected?

Currently hot, warm and cold directories are on the same type of storage - however initially, there was a requirement to move to cold directory, as this would be cheaper disk space. As this is not the case anymore we can neglect this requirement.

0 Karma

damode
Motivator

Hi @craigwilkinson ,

just curious to know, on what factor did you choose 6 for maxWarmDBcount ? is it the number of days ?
because I am facing the same issue. I had set maxHotSpanSecs = 2592000 [hot bucket - 30 days] and still hot bucket didnt roll to cold. I am not sure what maxWarmDBcount in this case.

0 Karma

craigwilkinson
Path Finder

Hey @damode,

Apologies, I think there was an error with my initial configuration.

We're currently running maxWarmDBCount=3.

I'm not 100% sure what the reasoning was as this was a year ago sorry.
But setting this value above 1 addressed the issue of buckets not rolling.

-Craig

0 Karma

damode
Motivator

Thanks, @craigwilkinson.

0 Karma

pmalcakdoj
Path Finder

When maxHotBuckets=1, maxHotSpanSecs is ignored.

NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all events to the single hot bucket and maxHotSpanSeconds will not be enforced.

Because of this, hot bucket will now only be rolled due to size (ie. 400000MB in your case)

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...