Splunk Search

Difference between maxHotIdleSecs and maxHotSpanSecs

strive
Influencer

Hi,

What is the difference between maxHotIdleSecs and maxHotSpanSecs. After reading the documentation i understood that both are used to roll data from hot to warm.

I read splunk documentation on indexes.conf and the link http://wiki.splunk.com/Deploy:BucketRotationAndRetention

I would like to know the exact difference between these options.

Thanks

Strive

Tags (1)
1 Solution

dwaddle
SplunkTrust
SplunkTrust

Adding to Kristian's answer, these two settings are somewhat complimentary.

The "span" of a bucket is the difference between the lowest _time and highest _time in a bucket. The value of maxHotSpanSecs is useful for controlling the amount of "time" in a bucket. For a quickly growing bucket, this is almost unimportant - the bucket will exceed the size parameter well before it exceeds the time parameter.

Splunk has advised in the past that a few larger buckets is better than many smaller ones. But, smaller buckets can be useful for controlling the size (technically time range) of your data retention policy (using frozenTimePeriodInSecs). When you set frozenTimePeriodInSecs, Splunk will only freeze (delete) a bucket when the newest event is older than that. So, with a maxHostSpanSecs of 90 days and a frozenTimePeriodInSecs of 90 days, you could have events up to 179 days old in the oldest bucket of a slow index. This is an issue for some people.

I would assume the same as Kristian for maxHotIdleSeconds that it is the difference between the newest event in the bucket and 'now'.

But, for both of these options, I would probably leave them alone and stick to defaults -- letting Splunk pick an appropriate bucket size and roll as needed.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

Adding to Kristian's answer, these two settings are somewhat complimentary.

The "span" of a bucket is the difference between the lowest _time and highest _time in a bucket. The value of maxHotSpanSecs is useful for controlling the amount of "time" in a bucket. For a quickly growing bucket, this is almost unimportant - the bucket will exceed the size parameter well before it exceeds the time parameter.

Splunk has advised in the past that a few larger buckets is better than many smaller ones. But, smaller buckets can be useful for controlling the size (technically time range) of your data retention policy (using frozenTimePeriodInSecs). When you set frozenTimePeriodInSecs, Splunk will only freeze (delete) a bucket when the newest event is older than that. So, with a maxHostSpanSecs of 90 days and a frozenTimePeriodInSecs of 90 days, you could have events up to 179 days old in the oldest bucket of a slow index. This is an issue for some people.

I would assume the same as Kristian for maxHotIdleSeconds that it is the difference between the newest event in the bucket and 'now'.

But, for both of these options, I would probably leave them alone and stick to defaults -- letting Splunk pick an appropriate bucket size and roll as needed.

kristian_kolb
Ultra Champion

from the indexes.conf docs page;

maxHotSpanSecs = <positive integer>
    * Upper bound of timespan of hot/warm buckets in seconds.
    * Defaults to 7776000 seconds (90 days).
    * NOTE: If you set this too small, you can get an explosion of hot/warm
      buckets in the filesystem.
    * This parameter cannot be set to less than 3600; if you set it to a lesser
      value, it will be automatically reset to 3600, which will then activate
      snapping behavior (see below).
    * This is an advanced parameter that should be set
      with care and understanding of the characteristics of your data.
    * If set to 3600 (1 hour), or 86400 (1 day), becomes also the lower bound
      of hot bucket timespans.  Further, snapping behavior (i.e. ohSnap)
      is activated, whereby hot bucket boundaries will be set at exactly the hour
      or day mark, relative to local midnight.
    * Highest legal value is 4294967295

maxHotIdleSecs = <positive integer>
    * Maximum life, in seconds, of a hot bucket.
    * If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm.
    * This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.
    * A value of 0 turns off the idle check (equivalent to infinite idle time).
    * Defaults to 0.
    * Highest legal value is 4294967295

Hm, I agree that it's somewhat confusing. Just from the naming of the parameters, I could imagine that maxHotIdleSecs should really be measuring (in seconds) from the last/latest event that entered a hot bucket, whereas maxHotSpanSecs would measure from the first/earliest event in a bucket.

These are just my guesses, and I also see that it says "Maximum life, in seconds, of a hot bucket" for maxHotIdleSecs. That could be an error/typo in the docs... but I have not really done any empirical testing of this.

/K

Get Updates on the Splunk Community!

Happy CX Day to our Community Superheroes!

Happy 10th Birthday CX Day!What is CX Day? It’s a global celebration recognizing innovation and success in the ...

Check out This Month’s Brand new Splunk Lantern Articles

Splunk Lantern is a customer success center providing advice from Splunk experts on valuable data insights, ...

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...