Hi Team,
There was a recent failure on one of our host where we have hosted Splunk App for DB Connect.Resultant we have lost data for almost 5,6 hours.
I wanted to create a secondary backup server, if something goes wrong with our present server, Ideally it should start working on the secondary server.
Also, i have realized that decrypting the passwords and entering it with script are not working, last time i tried I had to put it manually all the passwords.
Could anyone please guide me or gives me the direction how can i achieve that ?
Regards,
Complete failover automation is impossible.
but you can have workaround.
Splunk manual start is required on secondary Splunk instance where Splunk_app_dbconnect will be idle.
1. open firewall rules from secondary instance to all destinations from which you are collecting records from active instance.
2. Configure all inputs on secondary instance
3. don’t start splunk service
4. rsync db checkpoints available at $SPLUNK_HOME/var/lib/splunk/modinputs/splunk_app_dbconnect/* for every one minute from active to secondary instance
5. Configure to get an alert if primary Instance Splunkd service is down.
6. You can have simple alert action which will be called by the alert.
you have to test this before taking decisions.
this is my idea only. Not tested really.
I ran into same situation today. I followed same steps given in the previous answer and they worked as expected.