How to troubleshoot error "idx=... Throttling indexer, too many tsidx files in bucket=..." on my distributed management console?

[a212830]

Hi,

I noticed this popping up on my Distributed Management Console the other day:

throttled - idx=ngccc_app_logs Throttling indexer, too many tsidx files in bucket='/apps/splunk/db/ngccc_app_logs/db/hot_v1_20299'. Is splunk-optimize working? If not, low disk space may be the cause. 

My systems have lots of space (2+tb) available. Never seen this one before - any suggestions? It's a high volume index.

[swmishra_splunk]

Kindly, check for which specific indexes and for which bucket directories it is giving the error.

Generally, whenever an index generates too many small tsidx files(more than 25) Splunk is not able to optimize all those files within the specified time period.

Kindly, run the below command against the specific directory to optimize it manually:-

splunk-optimize -d|--directory

Or you can make the below changes in Indexes.conf to fix the issue:-

indexes.conf
[default]
maxConcurrentOptimizes=25
maxRunningProcessGroups=12
processTrackerServiceInterval=0

Please go through the below documentation to have a better understanding of Splunk Optimization.
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes

[rickspartan]

@swmishra_splunk hi, we are seeing the same "too many tsidx files" on our DMC - for indexers that are overloaded on CPU, which half is splunk-optimizes. I've seen up to 164 tsidx files in one bucket and our optimize settings are default. Version 8.2.2.1. These are on hardware servers with 16 cores.

maxConcurrentOptimizes = 6
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
processTrackerServiceInterval = 1

Our main issue is that we're running into ingest lag due to CPU being saturated by the optimize processes, along with search/ingest. I was suggested the we try setting the processTrackerServiceInterval to 60 to allow ingest lag to catch up. Will that help us there? I will try matching the other settings you recommend as well, but wanted to test 1 thing at a time.

[swmishra_splunk]

If you are getting the error only for this specific directory you can run Splunk Optimization manually by the below command:-

splunk-optimize -d|--directory

But If you are getting multiple errors for different directories then It might be a bug.

In that case First, check what is the Splunk Version you are running on the Server.

It is a bug in older versions of Splunk(till 7.0.4.) and was fixed from 7.0.5.onwards.

If it is a lower version of Splunk you can upgrade Splunk to a higher version or you can make the below changes in your Indexes.conf to fix the issue.

indexes.conf

[default]
maxConcurrentOptimizes=25
maxRunningProcessGroups=12
processTrackerServiceInterval=0

[bsachitano]

Check this thread out.
https://answers.splunk.com/answers/110173/search-peer-has-the-following-message-idx-internal-throttl...