How to Protect the Master Node in an Index Cluster

[FritzWittwer_ol]

We are running a two-site index cluster with three indexers on each site. We plan to have a standby master node (replication master) on the second site. Can we have a DNS alias with a list of two nodes, the active and the standby replication master in the server.conf of the slave Indexers and the search heads?

Thus, we would only need to guarantee that never both masters are running at the same time, but we do not need to change any configuration setting on the indexers or search heads. Using the same IP is not an option, as we have no layer 2 connection between the two sites.

Or are there other options, except load balancers, to failover the replication master to the other site while there is no layer two Connection.

[Steve_G_]

This issue is discussed in the following topic: http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Handlemasternodefailure

[ckurtz]

I would highly suggest avoiding having two IPs listed for the A record of the Cluster Master. Every time an indexer or searchhead tried to go to the one that was down you'd have issues.

Instead, I would have a single A record with a very short TTL, so it's easy to switch to the backup Cluster Master if needed by changing DNS.

For syncing between the Primary and Backup Cluster Masters I'd use either rsync or better a version control system (git, subversion) and do automated checkins/checkouts.