The cluster master and cluster peers are fully set up and were successfully communicating. After upgrading all instances to 5.0.4, the cluster master is no longer able to successfully move configuration in cluster-apps on the cluster master to slave-apps on the cluster peers.
Receive the following errors in splunkd.log:
08-20-2013 13:35:43.529 -0400 ERROR CMSlave - Failed to move bundle to slave-apps
host=splunk-indexer6 Options| sourcetype=splunkd Options| source=/opt/splunk/var/log/splunk/splunkd.log
08-20-2013 13:35:43.528 -0400 ERROR CMSlave - Could not move /opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/_cluster to /opt/splunk/etc/slave-apps/_cluster
In our case:
Our directory $SPLUNK_HOME/var was located on another partition than $SPLUNK_HOME/etc and was referenced by a symlink.
Splunk was not able to move the bundle directory from one partition to another (/opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/_cluster to /opt/splunk/etc/slave-apps/_cluster).
Splunk considered this behaviour as a bug in 5.0.4 and planned to fix it.