We have search head clustering and index clustering (not multisite) enabled on our environment.
We have splunk db connect v1 installed on our deployment server and the deployment server: /apps/splunk/etc/system/local/outputs.conf has
We have a db connect input (tail) setup on the deployment server which is set to send event to a specific index. It was working till yesterday and suddenly not sending any data to the index.
I have verified that /apps/splunk/var/lib/splunk/persistentstorage/dbx/6ef870be8f52c4ff7a8c4d303e193ce4/state.xml is being updated regularly with new rising column value. however events are not getting updated in indexers.
something i found in the /Splunk/splunk/var/log/splunk/metrics.log of indexers are,
Make sure to include details on any recent changes to the environment.
Validate if the DS is sending ANY data to the indexers.
Also, make sure the DS is not over-committed as it takes most of its resources for DS activity. As such, you might consider a Data Collection tier which is a collection of Forwarders (Heavy or Universal - depending on needs) dedicated for this type of activity.