Getting Data In
Highlighted

How to resolve error message after indexer went down: "too many tsidx files in bucket"?

Ultra Champion

One indexer just went down. As it came up we see the following message for a couple of the indexers -

throttled: idx=<idx_name> Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes/<idx_name>/db/hot_v1_1519'. Is splunk-optimize working? If not, low disk space may be the cause. 

What it is exactly?

0 Karma
Highlighted

Re: How to resolve error message after indexer went down: "too many tsidx files in bucket"?

Splunk Employee
Splunk Employee

The below errors indicates "high I/O activity of splunk-optimize":

throttled: idx= Throttling indexer, too many tsidx files in bucket='/SplunkIndexData/splunk-indexes//db/hotv11519'. Is splunk-optimize working? If not, low disk space may be the cause.

The way to reduce splunk-optimize I/O activity is to increase maxMemMB for index "" which has most of throttling messages and any other index customer is aware of high volume:

indexes.conf
[index_name]
maxMemMB=20

Highlighted

Re: How to resolve error message after indexer went down: "too many tsidx files in bucket"?

Splunk Employee
Splunk Employee

Additional to that if you see the below ERROR as well you can also increase the value of maxConcurrentOptimizes in indexes.conf for a particular index which is affected or you can set globally to all the indexes depends on your requirement.

ERROR:
04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child39286SplunkOptimize) optimize finished: failed, see rc for more details, dir=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hotv1_268, rc=-12 (unsigned 244), errno=0

04-11-2019 11:01:51.574 +0500 ERROR SplunkOptimize - (child39286SplunkOptimize) merge failed for path=C:\Program Files\Splunk\var\lib\splunk_internaldb\db\hotv1268 rc=-12 wrc=-12 errno=0 file=dontknow hint=tsvalid is UINTMAX in _mergeall_postings]

Example:

In indexes.conf

For particular index:
[_internal]
maxConcurrentOptimizes = < desired value >

For all the indexes:
[default]
maxConcurrentOptimizes = < desigered value >

maxConcurrentOptimizes = < nonnegative integer >

  • The number of concurrent optimize processes that can run against the hot DB.
  • This number should be increased if:
    • There are always many small tsidx files in the hot DB.
    • After rolling, there are many tsidx files in warm or cold DB.
  • Must restart splunkd after changing this parameter; index reload will not suffice.
  • Highest legal value is 4294967295
  • Defaults to 6
0 Karma