Too many tsidx files

[rxdeleon]

I keep on getting the following message. Which "disk space" is this message referring to? The /var/splunk filesystem currently has 20 GB of free space.

"applying indexing throttle for /var/splunk/mck-app-log/db because bucket has too many tsidx files, is your splunk-optimize working? splunk-optimize may be prevented from running if the minimum disk space usage limit is reached."

[saramamurthy_sp]

Kindly, check for which specific indexes and for which bucket directories it is giving the error.

Generally, whenever an index generates too many small tsidx files(more than 25) Splunk is not able to optimize all those files within the specified time period.

Kindly, run the below command against the specific directory to optimize it manually:-

splunk-optimize -d|--directory

Or you can make the below changes in Indexes.conf to fix the issue:-

indexes.conf
[default]
maxConcurrentOptimizes=25
maxRunningProcessGroups=12
processTrackerServiceInterval=0

Please go through the below documentation to have a better understanding of Splunk Optimization.
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Optimizeindexes

[araitz]

This can happen if you are indexing a lot of data at once. If you look in the above directory, how many *.tsidx files do you see?

In rare cases where you don't have much CPU and/or disk IOPS, splunk-optimize.exe might take a long time to run and it could fall behind. If you use top on linux or task manager on Windows, do you see that any splunk-optimize or splunk-optimize.exe processes have been running for more than a minute or two?

[araitz]

Sorry for not asking specfically before, but how many tsidx files per bucket (e.g. db_* or 'hot_*`)? A few up to a few dozen per bucket is fine. 300 per bucket is not. If it is the latter, what are the specs of your system (OS, filesystem, CPU, Memory, Disk) and how much data are you indexing per day (approximately)?

[rxdeleon]

I see almost 300 *.tsidx files but I don't see splunk-optimize running. Is there a log file where splunk-optimize writes to?