One indexer just went down. As it came up we see the following message for a couple of the indexers -
throttled: idx=<idx_name> Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes/<idx_name>/db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause.
What it is exactly?
The below errors indicates "high I/O activity of splunk-optimize":
throttled: idx= Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes//db/hotv11519'. Is splunk-optimize working? If not, low disk space may be the cause.
The way to reduce splunk-optimize I/O activity is to increase maxMemMB for index "" which has most of throttling messages and any other index customer is aware of high volume:
Additional to that if you see the below ERROR as well you can also increase the value of maxConcurrentOptimizes in indexes.conf for a particular index which is affected or you can set globally to all the indexes depends on your requirement.
04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child39286SplunkOptimize) optimize finished: failed, see rc for more details, dir=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hotv1_268, rc=-12 (unsigned 244), errno=0
04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child39286SplunkOptimize) merge failed for path=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hotv1268 rc=-12 wrc=-12 errno=0 file=dontknow hint=tsvalid is UINTMAX in _mergeall_postings]
For particular index:
maxConcurrentOptimizes = < desired value >
For all the indexes:
maxConcurrentOptimizes = < desigered value >
maxConcurrentOptimizes = < nonnegative integer >