Indexing performance has started to become really bad to the point where my Splunk instance has stopped indexing events or at least appears to have stopped doing so. Restarting Splunk does not resolve the problem and I have not made any changes to my working configuration.
There are plenty of system resources available.
The only thing that seem out of the ordinary is that the Sources.data file for some of my indexes seems somewhat large (5MB - 140MB)
Could this be causing an issue? If so, what can I do to bring my indexing back to an acceptable level?
You are correct, the large global metadata files are the culprit.
This is the global metadata file for the Sources, stored in the root folder of each index.
If a file is larger than 50 MB you will start to notice performance impact.
The consequence is slow indexing speed, because of the manipulation of those large file at index time.
The impacted files depend of the type of data you are receiving.
by example for the main index, the files are like :
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/Sources.data
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/SourceTypes.data
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/Hosts.data
The simple workaround is to upgrade to the splunk version 5.* or more recent, because this global metadata feature has been removed since 5.* (see in indexes.conf disableGlobalMetadata=true)
Important remarks :
You are correct, the large global metadata files are the culprit.
This is the global metadata file for the Sources, stored in the root folder of each index.
If a file is larger than 50 MB you will start to notice performance impact.
The consequence is slow indexing speed, because of the manipulation of those large file at index time.
The impacted files depend of the type of data you are receiving.
by example for the main index, the files are like :
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/Sources.data
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/SourceTypes.data
$SPLUNK_HOME/var/lib/splunk/defaultdb/db/Hosts.data
The simple workaround is to upgrade to the splunk version 5.* or more recent, because this global metadata feature has been removed since 5.* (see in indexes.conf disableGlobalMetadata=true)
Important remarks :
Changing the sources or sourcetypes, will prevent the issue to grow on the long term , but not improve the indexing speed.
Except if you can roll the previous buckets to frozen.
It would probably help only if you rotated the buckets with big *.data files to frozen (thus removing them from searchable data)
Would it help if I reduce the number of sources, specify the sourcetype and the host for my inputs?