Based on inputs from Splunk PS and testing, in the SHP environment the inputs.conf for DBConnect should be on $SPLUNK_HOME/etc /system/local area on ONLY 1 SH.
When I had asked this question, I had already put inputs.conf with just dbmon-tail in the local area and restarted splunk, however the inputs.conf was not getting read and the above messages were getting logged. After several retries, when this did not work, I added another input - with dbmon-dump. Once I did that and restarted splunk, suddenly inputs.conf in system/local area started getting read and DBconnect started working - HOW -WHY, I don't know.
I had opened a ticket with support to validate this, however per them inputs.conf for DB connect outside pooled area is not supported but they were also unable to tell me how to configure DB Connect based on documentation statement: "While you can install and configure Splunk DB Connect on a single search head, in a pooling environment, the app state is written to shared storage and is visible to all search heads. ". and kept referring to the Architecture statement:
`About search head pooling and dbmon-tail
We do not recommend using dbmon-tail inputs in a search head pooling environment. In a search head pooling environment, each search head has its own persistent storage that keeps track of the last rising column. This can cause Splunk to index different values for each search head.
We recommend instead that you use a dedicated heavy forwarder with DB Connect installed, to forward data to Splunk indexers.`
Anyways, at this point DB connect has been working for us over a month as expected in the SHP environment and I have not seen any performance issue on any of the SH's.
... View more