Cluster Master unable to send bundles to slave-apps directory on cluster peers

[katelynengel]

The cluster master and cluster peers are fully set up and were successfully communicating. After upgrading all instances to 5.0.4, the cluster master is no longer able to successfully move configuration in cluster-apps on the cluster master to slave-apps on the cluster peers.

Receive the following errors in splunkd.log:

08-20-2013 13:35:43.529 -0400 ERROR CMSlave - Failed to move bundle to slave-apps
host=splunk-indexer6 Options| sourcetype=splunkd Options| source=/opt/splunk/var/log/splunk/splunkd.log

08-20-2013 13:35:43.528 -0400 ERROR CMSlave - Could not move /opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/_cluster to /opt/splunk/etc/slave-apps/_cluster

[icssupport]

We already checked the permissions settings. We have another problem. But we are in contact with splunk support ...

[icssupport]

In our case:
Our directory $SPLUNK_HOME/var was located on another partition than $SPLUNK_HOME/etc and was referenced by a symlink.
Splunk was not able to move the bundle directory from one partition to another (/opt/splunk/var/run/splunk/cluster/remote-bundle/72329e457bf34d9351eb50a4dec40a85-1377020099/apps/_cluster to /opt/splunk/etc/slave-apps/_cluster).
Splunk considered this behaviour as a bug in 5.0.4 and planned to fix it.

[davidpaper]

What was the solution?

[mahamed_splunk]

ICSSupport, Are you getting the same error message? It means that the directory _cluster has restricted permission. Could you check the permission settings of that directory?

[icssupport]

May I ask if this has already been solved and if yes, what was the solution?

[mahamed_splunk]

Could you check the permission on the _cluster directory?