What is the cause of this buckets error?

[marka3721]

Have a index that is throwing up a warning, and the Root Cause says The newly created warm bucket size is too large. The bucket size=32630820864 exceeds the yellow_size_threshold=20971520000 from the latest_detected_index. This index was created just all the other indexes, and this one is the only one that is throwing the warning. And there has been at least 6 months of data be sent to this index, and it is saying there is only 14 days of data. What could be the issue with this index.

[shivanshu1593]

The problem lies in the following. Your maxDataSize (Size of a hot bucket is 307.2 GB as per your config) is larger than your index size which is governed by maxTotalDataSizeMB (Currently set at 204.2 GB in your config), causing all kinds of mayhem. Please set it back to its default value, which is "auto" and push the indexes.conf to the indexer via cluster master (If it's in an indexer cluster). The data takes a while to get rolled off the hot bucket to warm, which is why for high volume index you will find that the maximum size of your warm buckets might exceed hot buckets size and throw errors like you are seeing. As a general thumb rule, always remember that maxTotalDataSizeMB > maxDataSize. Since this seems to be firewall data, hope you have considered the volume of data to be expected while setting maxTotalDataSizeMB to 204 GB. Once it gets more data, the oldest log file will start to get deleted.

maxDataSize = 307200
maxTotalDataSizeMB = 204800

++If this helps, please consider accepting as an answer++

[shivanshu1593]

Can you shared the content of your indexes.conf for the said index, masking the sensitive information but not renaming any of the actual configs already present in Splunk docs to help diagnose this further?

[marka3721]

This is the index config that is throwing the warnings

[fortinet]
coldPath = $SPLUNK_DB/fortinet/colddb
enableDataIntegrityControl = 0
enableTsidxReduction = 0
homePath = $SPLUNK_DB/fortinet/db
maxDataSize = 307200
maxTotalDataSizeMB = 204800
thawedPath = $SPLUNK_DB/fortinet/thaweddb
archiver.enableDataArchive = 0
bucketMerging = 0
bucketRebuildMemoryHint = 0
compressRawdata = 1
enableOnlineBucketRepair = 1
hotBucketStreaming.deleteHotsAfterRestart = 0
hotBucketStreaming.removeRemoteSlicesOnRoll = 0
hotBucketStreaming.reportStatus = 0
hotBucketStreaming.sendSlices = 0
metric.enableFloatingPointCompression = 1
metric.stubOutRawdataJournal = 1
minHotIdleSecsBeforeForceRoll = 0
rtRouterQueueSize =
rtRouterThreads =
selfStorageThreads =
suspendHotRollByDeleteQuery = 0
syncMeta = 1
tsidxWritingLevel =