Why Splunkd daemon not responding after distributing configuration bundle with master node?

[scc00]

I have the following configuration: 1 Master Node, 1 Search Head, 2 Peers running version 6.1.1. RF 2, SF 1. When I update the indexes.conf within */master_apps/_cluster on the master node and then attempt to Distribute the Configuration Bundle to the peers, the master node disconnects and acts as though it is restarting. Any thoughts? Within the splunkd.log it shows the following: Note, I have rebooted the master node and ensured the peers were all online, the distribution bundle does not take.

Splunkd daemon is not responding: ('Error connecting to /services/cluster/master/control/default/apply: The read operation timed out',)

Splunkd logs

07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~5~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~5~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~6~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~6~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~7~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~7~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~8~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - Apply bundle - all peers are down. Will not switch bundles.

[ofrachon]

Hello,

It looks like your Master Node does not "see" any of your peers.

It could be a configuration problem (i.e. secret key not properly shared) or a network problem.
Make sure the [clustering] stanza in the server.conf files on all members of the cluster (master node, search head, peers) are homogeneous.

Relevant stuff can be found on the Splunk Docs site : http://docs.splunk.com/Documentation/Splunk/6.1.2/Indexer/Clusterdeploymentoverview

Also, you should NOT put anything in $SPLUNK_HOME/etc/master-apps/_cluster as it is reserved for splunk stuff.

Instead you should put your apps in $SPLUNK_HOME/etc/master-apps/app1, $SPLUNK_HOME/etc/master-apps/app2 and so on.

Stuff regarding the Configuration bundle :
http://docs.splunk.com/Documentation/Splunk/6.1.2/Indexer/Updatepeerconfigurations

Regards.