We have a single server Splunk enterprise № 1, which receives data from the database using DB connect. We have another server Splunk enterprise № 2. Server Splunk № 1 sends events to Splunk № 2 using forwarding. When server № 2 becomes unavailable, events from DB connect cease to arrive at the server Splunk №1. After resuming the availability of the server number 2, everything is restored.
How to make independent server № 1 and server № 2?
Per my knowledge db connect is not built to work in high availability environment, this mean that if you deploy the db connect to different serves and keep the both of them enabled, the data will be indexed twice.
I did a solution to customer where the db connect was deployed in two separate heavy forwarder, but one of them remain disabled, so when the main heavy forwarder is down for any reason, the db connect on the stand by heavy forwarder have to be enabled manually and splunk service has to be restarted in order to continue indexing the data. The trick part here is you have to keep the both app update with the configuration. As the data is critical to the business I did not implement any automation to avoid the particular app to be enable by mistake and indexed the data twice. At the moment I don't have any other solution to share, I hope this helps you.