topic Re: How to resolve error message after indexer went down: "too many tsidx files in bucket"? in Getting Data In

How to resolve error message after indexer went down: "too many tsidx files in bucket"?

ddrillic — Tue, 03 Oct 2017 16:52:08 GMT

One indexer just went down. As it came up we see the following message for a couple of the indexers -

throttled: idx=<idx_name> Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes/<idx_name>/db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause.

What it is exactly?

Re: How to resolve error message after indexer went down: "too many tsidx files in bucket"?

mbagali_splunk — Wed, 30 Sep 2020 00:13:58 GMT

The below errors indicates "high I/O activity of splunk-optimize":

throttled: idx= Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes//db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause.

The way to reduce splunk-optimize I/O activity is to increase maxMemMB for index "" which has most of throttling messages and any other index customer is aware of high volume:

indexes.conf
[index_name]
maxMemMB=20

Re: How to resolve error message after indexer went down: "too many tsidx files in bucket"?

anaidu_splunk — Wed, 30 Sep 2020 00:14:00 GMT

Additional to that if you see the below ERROR as well you can also increase the value of maxConcurrentOptimizes in indexes.conf for a particular index which is affected or you can set globally to all the indexes depends on your requirement.

ERROR:
04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) optimize finished: failed, see rc for more details, dir=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268, rc=-12 (unsigned 244), errno=0

04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child_39286__SplunkOptimize) merge failed for path=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hot_v1_268 rc=-12 wrc=-12 errno=0 file=dontknow hint=tsval_id is UINT_MAX in _merge_all_postings]

Example:

In indexes.conf

For particular index:
[_internal]
maxConcurrentOptimizes = < desired value >

For all the indexes:
[default]
maxConcurrentOptimizes = < desigered value >

maxConcurrentOptimizes = < nonnegative integer >

The number of concurrent optimize processes that can run against the hot DB.
This number should be increased if:
- There are always many small tsidx files in the hot DB.
- After rolling, there are many tsidx files in warm or cold DB.
Must restart splunkd after changing this parameter; index reload will not suffice.
Highest legal value is 4294967295
Defaults to 6