Hi,
What is the difference between maxHotIdleSecs and maxHotSpanSecs. After reading the documentation i understood that both are used to roll data from hot to warm.
I read splunk documentation on indexes.conf and the link http://wiki.splunk.com/Deploy:BucketRotationAndRetention
I would like to know the exact difference between these options.
Thanks
Strive
Adding to Kristian's answer, these two settings are somewhat complimentary.
The "span" of a bucket is the difference between the lowest _time
and highest _time
in a bucket. The value of maxHotSpanSecs
is useful for controlling the amount of "time" in a bucket. For a quickly growing bucket, this is almost unimportant - the bucket will exceed the size parameter well before it exceeds the time parameter.
Splunk has advised in the past that a few larger buckets is better than many smaller ones. But, smaller buckets can be useful for controlling the size (technically time range) of your data retention policy (using frozenTimePeriodInSecs
). When you set frozenTimePeriodInSecs
, Splunk will only freeze (delete) a bucket when the newest event is older than that. So, with a maxHostSpanSecs
of 90 days and a frozenTimePeriodInSecs
of 90 days, you could have events up to 179 days old in the oldest bucket of a slow index. This is an issue for some people.
I would assume the same as Kristian for maxHotIdleSeconds
that it is the difference between the newest event in the bucket and 'now'.
But, for both of these options, I would probably leave them alone and stick to defaults -- letting Splunk pick an appropriate bucket size and roll as needed.
Adding to Kristian's answer, these two settings are somewhat complimentary.
The "span" of a bucket is the difference between the lowest _time
and highest _time
in a bucket. The value of maxHotSpanSecs
is useful for controlling the amount of "time" in a bucket. For a quickly growing bucket, this is almost unimportant - the bucket will exceed the size parameter well before it exceeds the time parameter.
Splunk has advised in the past that a few larger buckets is better than many smaller ones. But, smaller buckets can be useful for controlling the size (technically time range) of your data retention policy (using frozenTimePeriodInSecs
). When you set frozenTimePeriodInSecs
, Splunk will only freeze (delete) a bucket when the newest event is older than that. So, with a maxHostSpanSecs
of 90 days and a frozenTimePeriodInSecs
of 90 days, you could have events up to 179 days old in the oldest bucket of a slow index. This is an issue for some people.
I would assume the same as Kristian for maxHotIdleSeconds
that it is the difference between the newest event in the bucket and 'now'.
But, for both of these options, I would probably leave them alone and stick to defaults -- letting Splunk pick an appropriate bucket size and roll as needed.
from the indexes.conf
docs page;
maxHotSpanSecs = <positive integer>
* Upper bound of timespan of hot/warm buckets in seconds.
* Defaults to 7776000 seconds (90 days).
* NOTE: If you set this too small, you can get an explosion of hot/warm
buckets in the filesystem.
* This parameter cannot be set to less than 3600; if you set it to a lesser
value, it will be automatically reset to 3600, which will then activate
snapping behavior (see below).
* This is an advanced parameter that should be set
with care and understanding of the characteristics of your data.
* If set to 3600 (1 hour), or 86400 (1 day), becomes also the lower bound
of hot bucket timespans. Further, snapping behavior (i.e. ohSnap)
is activated, whereby hot bucket boundaries will be set at exactly the hour
or day mark, relative to local midnight.
* Highest legal value is 4294967295
maxHotIdleSecs = <positive integer>
* Maximum life, in seconds, of a hot bucket.
* If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm.
* This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.
* A value of 0 turns off the idle check (equivalent to infinite idle time).
* Defaults to 0.
* Highest legal value is 4294967295
Hm, I agree that it's somewhat confusing. Just from the naming of the parameters, I could imagine that maxHotIdleSecs
should really be measuring (in seconds) from the last/latest event that entered a hot bucket, whereas maxHotSpanSecs
would measure from the first/earliest event in a bucket.
These are just my guesses, and I also see that it says "Maximum life, in seconds, of a hot bucket" for maxHotIdleSecs
. That could be an error/typo in the docs... but I have not really done any empirical testing of this.
/K