How to resolve error message after indexer went down: "too many tsidx files in bucket"?

[ddrillic]

One indexer just went down. As it came up we see the following message for a couple of the indexers -

throttled: idx=<idx_name> Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes/<idx_name>/db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause. 

What it is exactly?

[anaidu_splunk]

Additional to that if you see the below ERROR as well you can also increase the value of maxConcurrentOptimizes in indexes.conf for a particular index which is affected or you can set globally to all the indexes depends on your requirement.

ERROR:
04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) optimize finished: failed, see rc for more details, dir=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268, rc=-12 (unsigned 244), errno=0

04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) merge failed for path=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268 rc=-12 wrc=-12 errno=0 file=dontknow hint=tsval_id is UINT_MAX in _merge_all_postings]

Example:

In indexes.conf

For particular index:
[_internal]
maxConcurrentOptimizes = < desired value >

For all the indexes:
[default]
maxConcurrentOptimizes = < desigered value >

maxConcurrentOptimizes = < nonnegative integer >

• The number of concurrent optimize processes that can run against the hot DB.
• This number should be increased if:
  • There are always many small tsidx files in the hot DB.
  • After rolling, there are many tsidx files in warm or cold DB.
• Must restart splunkd after changing this parameter; index reload will not suffice.
• Highest legal value is 4294967295
• Defaults to 6

[mbagali_splunk]

The below errors indicates "high I/O activity of splunk-optimize":

throttled: idx= Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes//db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause.

The way to reduce splunk-optimize I/O activity is to increase maxMemMB for index "" which has most of throttling messages and any other index customer is aware of high volume:

indexes.conf
[index_name]
maxMemMB=20