Deployment Architecture

How to troubleshoot error "idx=... Throttling indexer, too many tsidx files in bucket=..." on my distributed management console?

a212830
Champion

Hi,

I noticed this popping up on my Distributed Management Console the other day:

throttled - idx=ngccc_app_logs Throttling indexer, too many tsidx files in bucket='/apps/splunk/db/ngccc_app_logs/db/hot_v1_20299'. Is splunk-optimize working? If not, low disk space may be the cause. 

My systems have lots of space (2+tb) available. Never seen this one before - any suggestions? It's a high volume index.

0 Karma

swmishra_splunk
Splunk Employee
Splunk Employee

Kindly, check for which specific indexes and for which bucket directories it is giving the error.

Generally, whenever an index generates too many small tsidx files(more than 25) Splunk is not able to optimize all those files within the specified time period.

Kindly, run the below command against the specific directory to optimize it manually:-

splunk-optimize -d|--directory

Or you can make the below changes in Indexes.conf to fix the issue:-

indexes.conf
[default]
maxConcurrentOptimizes=25
maxRunningProcessGroups=12
processTrackerServiceInterval=0

Please go through the below documentation to have a better understanding of Splunk Optimization.
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes

rickspartan
New Member

@swmishra_splunk hi, we are seeing the same "too many tsidx files" on our DMC - for indexers that are overloaded on CPU, which half is splunk-optimizes. I've seen up to 164 tsidx files in one bucket and our optimize settings are default. Version 8.2.2.1. These are on hardware servers with 16 cores.

maxConcurrentOptimizes = 6
maxRunningProcessGroups = 8
maxRunningProcessGroupsLowPriority = 1
processTrackerServiceInterval = 1

 

Our main issue is that we're running into ingest lag due to CPU being saturated by the optimize processes, along with search/ingest. I was suggested the we try setting the processTrackerServiceInterval to 60 to allow ingest lag to catch up. Will that help us there? I will try matching the other settings you recommend as well, but wanted to test 1 thing at a time.

0 Karma

swmishra_splunk
Splunk Employee
Splunk Employee

If you are getting the error only for this specific directory you can run Splunk Optimization manually by the below command:-

splunk-optimize -d|--directory

But If you are getting multiple errors for different directories then It might be a bug.

In that case First, check what is the Splunk Version you are running on the Server.

It is a bug in older versions of Splunk(till 7.0.4.) and was fixed from 7.0.5.onwards.

If it is a lower version of Splunk you can upgrade Splunk to a higher version or you can make the below changes in your Indexes.conf to fix the issue.

indexes.conf

[default]
maxConcurrentOptimizes=25
maxRunningProcessGroups=12
processTrackerServiceInterval=0

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...