The cluster master and cluster peers are fully set up and were successfully communicating. After upgrading all instances to 5.0.4, the cluster master is no longer able to successfully move configuration in cluster-apps on the cluster master to slave-apps on the cluster peers.
Receive the following errors in splunkd.log:
08-20-2013 13:35:43.529 -0400 ERROR CMSlave - Failed to move bundle to slave-apps
host=splunk-indexer6 Options| sourcetype=splunkd Options| source=/opt/splunk/var/log/splunk/splunkd.log
08-20-2013 13:35:43.528 -0400 ERROR CMSlave - Could not move /opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/cluster to /opt/splunk/etc/slave-apps/cluster
ICSSupport, Are you getting the same error message? It means that the directory _cluster has restricted permission. Could you check the permission settings of that directory?
We already checked the permissions settings. We have another problem. But we are in contact with splunk support ...
In our case:
Our directory $SPLUNKHOME/var was located on another partition than $SPLUNKHOME/etc and was referenced by a symlink.
Splunk was not able to move the bundle directory from one partition to another (/opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/cluster to /opt/splunk/etc/slave-apps/cluster).
Splunk considered this behaviour as a bug in 5.0.4 and planned to fix it.