Why are the buckets not rolling as per configuration under indexes.conf?

[dkolekar_splunk]

As per our configuration data retain is 730 days. maxHotSpanSecs is 30 days but we are noticing bucket with start date 1970 and end date 20190810

I can see a few buckets are in the warm state as well which has the earliest time as 1970 and latest as 2019. Kindly suggest why these buckets are available and how to delete unwanted buckets.

[dkolekar_splunk · Accepted solution]

While checking the cause of this issue, I found the following the log in splunkd.log

02-23-2019 19:54:18.228 +0000 INFO HotBucketRoller - finished moving hot to warm bid=indexname~17~C4C49C84-24A5-4BF4-9B2C-21AB428C447E idx=indexname from=hot_quar_v1_17 to=db_1570730892_0_17_C4C49C84-24A5-4BF4-9B2C-21AB428C447E size=86269952 caller=lru maxHotBuckets=3, count=3 hot buckets + 1 quar bucket,evicting_count=1 LRU hots

Which confirms that these buckets are quarantine buckets. The difference is that hot_v1_ is a normal hot bucket & hot_quar_v1_ is a quarantine bucket. Quarantine buckets are used to catch data that is older than quarantinePastSecs specified in indexes.conf or newer than quarantineFutureSecs specified by indexes.conf.

http://docs.splunk.com/Documentation/Splunk/latest/admin/indexesconf
quarantinePastSecs =
- Events with timestamp of quarantinePastSecs older than "now" will be
dropped into quarantine bucket.
- Defaults to 77760000 (900 days).
- This is a mechanism to prevent the main hot buckets from being polluted with
fringe events.

quarantineFutureSecs =
- Events with timestamp of quarantineFutureSecs newer than "now" will be
dropped into quarantine bucket.
- Defaults to 2592000 (30 days).
- This is a mechanism to prevent main hot buckets from being polluted with
fringe events.

1. The quarantine buckets store the data older than quarantinePastSecs parameter in indexes.conf and newer than "now" as per quarantinePastSecs.
2. Due to which we were seeing the older data and recent data in those buckets.
3. The number of quarantine buckets depends on maxHotBuckets parameter in indexes.conf maxHotBuckets = Quarantine buckets+ hot buckets per indexer and per index
4. In the above image, in total, you can see 12 hot buckets from 4 indexers. Which includes quarantine buckets as well as hot buckets.
5. The quarantine buckets are only available in the hot bucket stage.

Q. Is there a way to separate a way to put future events in the bucket and past events in another bucket?
A. No, this feature is not available in splunk. If we do so, the number of buckets will get increased and you will end up with multiple small buckets.

Q. How to remove unwanted buckets from splunk?
A. Buckets in the thawed state can be removed while the server is running, but hot/warm/ cold should not be. So, you need to
1. stop splunk
2. delete the offending bucket
3. start splunk