Indexer saturation

[Ethil]

[UPDATE]

Hello everyone, and thanks in advance for your help. I'm very new to this subject so if anything is unclear, i'll try to explain my problem more in details.

I'm using spunk 9.2.1, and i recently observed that my indexer was not indexing logs received. The indexer is in a failure state because my partition $SPLUNK_DB reached the minFreeSpace allowed in server.conf.

After further analysis it seems that one of the index _metrics on the partition is saturated with warm buckets (db_*) and taking all the space available. I however have configured all my indexes with the indexes.conf ($SPLUNK_HOME/etc/system/default/indexes.conf) 

# index specific defaults
maxTotalDataSizeMB = 5000
maxDataSize = 1000
maxMemMB = 5
maxGlobalRawDataSizeMB = 0
maxGlobalDataSizeMB = 0
rotatePeriodInSecs = 30
maxHotIdleSecs = 432000
maxHotSpanSecs = 7776000
maxHotBuckets = auto
maxWarmDBCount = 300
frozenTimePeriodInSecs = 188697600
...
# there's more but i might not be able to disclose them or it might not be revelant

[_metrics]
coldPath = $SPLUNK_DB/_metrics/colddb
homePath = $SPLUNK_DB/_metrics/db
thawedPath = $SPLUNK_DB/_metrics/thaweddb
frozenTimePeriodInSecs = 1209600

From what i understand with this conf applied the index should not exceed 5GB, and when reached the warm/hot buckets should be removed, but it seems that's it's not taken into account in my case. 

The indexer work fine after purging the buckets and restarting it, but i don't get why the conf was not applied ? Is there something i didn't get here ?

Is there a way to check the "characteristics" of my index once started ? -> Checked, the conf is correctly applied.

If you know anything on this subject please help me 🙂

thank you

[Ethil]

@PickleRick 

Ok i see, and yes currentDBSizeMB does correspond to the actual size of the index on the disk.

[PickleRick]

OK. I re-read your config and there is more going on underneath than meets the eye.

Firstly, Splunk index size management is not a precise thing.

Secondly, the bucket rolling happens on specific conditions. Most importantly - hot buckets do _not_ roll because of size restrictions on the index. So hot buckets will roll only when they meet hot bucket rolling criteria (too big bucket, too long idle period and such). So if you have a metric index, your buckets will probably grow to the maximum permissible size (which in your case is 1GB per bucket) and then some (Splunk adds some stuff when it closes the bucket and so on). Then it will get rolled to warm. And then it can pretty immediately get rolled directly to frozen if needed but no earlier.

Thirdly, you have maxHotBuckets=auto and you didn't redefine metric.maxHotBuckets, which by default is also "auto". That means Splunk will create 6 hot buckets for that index.

So your index will happily grow to at least 6GB regardless of your maxTotalDataSizeMB.

If you're way over that value (like you've reached some 20 or 30GB), it might be worth troubleshooting it with support since generally it shouldn't happen.

Oh, and hot buckets roll on indexer restart so it's natural that when you restart your indexer your disk usage goes down.

[Ethil]

@PickleRick ,

Sorry for the late answer, you are rigth, i think we might have misunderstood how some attributes work in the indexes.conf and thus it was not strong enough to force the rolling of the warm buckets. We will surely rework the conf and see what happens but i think that was the main issue.

Thanks a lot for your time and answers !

[PickleRick]

And you checked your effective settings with btool?

[richgalloway]

There is nothing technically wrong with the current setting.  Warm buckets did not roll to cold because none of the criteria for rolling buckets were met.  Reaching the minimum disk space is not a criterium.  Buckets roll either because the index is too full, the bucket(s) are too old, or the maximum number of warm buckets has been reached.

[Ethil]

Ok, but the indexes are all set with a maxTotalDataSIze of 5GB (default set up written in my indexes.conf), which from what i understood should have stop each indexes, individually, exceeding this size and force the older warm buckets to cold to avoid saturation.  

The doc :  https://docs.splunk.com/Documentation/Splunk/9.3.2/Admin/Indexesconf 

maxTotalDataSizeMB = <nonnegative integer>
* The maximum size of an index, in megabytes.
* If an index grows larger than the maximum size, splunkd freezes the oldest
  data in the index.
* This setting applies only to hot, warm, and cold buckets. It does
  not apply to thawed buckets.
...

However the saturation dit happen with one of them, that is the issue i don't understand. My disk is 40GB, and the saturation of this specific index reached 35GB and thus reached the minimum disk space and thus failed my indexer. The rolling criteria was met, why didn't it rolled the buckets ?