we have a multisite indexers cluster with two nodes and replication factor = 2. All instances are running Splunk 6.2. In the official documentation we read that it is not possible to back up just the data on a single node, since there's no certainty that a single node contains all the data in the cluster.
I suspect that this is not true in our scenario, where all data are replicated to both nodes (replication factor = total number of peers). Am I right? Can we backup data from one node (snapshot backup) and be sure that all data has been saved?
It would be nice to be able to check the cluster status before starting the backup job, raising a warning if it is not consistent. I think the only way to automate this check would be to run "bin/splunk show cluster-status --verbose" and check for "Replication factor met" and "Search factor met", but this command is available only on master node, while the check should be executed by the backup agent running on one of the peers. In addition, there would be an authentication request from Splunk daemon.
As a result, I think it is not possible to automate this check in a simple way.