Indexer Cluster - Cluster peer is downloading cluster bundle relatively

[sat94541]

User case:

 Cluster Masters with 30 Cluster Peers.
 Pushed the Cluster Bundle

 Most of the Cluster Peer Downloaded and applied the Bundle.
 Only few of the Cluster peer are unable to download and apply the cluster Bundle

=> CPs' splunkd.log shows it is repeatedly downloading and validation, restarting as follows

01-15-2015 17:45:50.294 -0800 INFO CMBundleMgr - bundle=/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/a4368cc2831176b43141074d64ce4c25-1421372747.bundle, checksum=8281284186D8D917A0DCD8366D9F5316 found on the slave
01-15-2015 17:45:50.315 -0800 INFO CMBundleMgr - Downloaded bundle path=/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/6cade09833f253046490922cde47c84f-1421372750.bundle time_taken_ms=21.
01-15-2015 17:45:50.315 -0800 INFO CMBundleMgr - untarring bundle=/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/6cade09833f253046490922cde47c84f-1421372750.bundle
01-15-2015 17:45:50.321 -0800 INFO ClusterBundleValidator - Validating bundle path=/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/6cade09833f253046490922cde47c84f-1421372750/apps
01-15-2015 17:45:50.339 -0800 INFO CMBundleMgr - Downloaded and validated the active bundle with id=C45E9654AEC18162AB09A1D4EAC45264
01-15-2015 17:45:50.339 -0800 INFO CMSlave - event=getActiveBundle status=success path=/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/6cade09833f253046490922cde47c84f-1421372750.bundle cksum=C45E9654AEC18162AB09A1D4EAC45264 alreadyin=0
01-15-2015 17:45:50.340 -0800 INFO CMBundleMgr - successfully moved bundle from='/home/test/splunk615CP/var/run/splunk/cluster/remote-bundle/6cade09833f253046490922cde47c84f-1421372750' to='/home/test/splunk615CP/etc/slave-apps'
01-15-2015 17:45:50.340 -0800 INFO CMBundleMgr - Remove old bundles.
01-15-2015 17:45:50.340 -0800 INFO loader - Downloaded new bundle from the cluster master. Restarting splunkd

[rbal_splunk]

Try these steps as it may help resolve this issue ...

1. Shutdown the peer
   1. Deleting the slave-apps and slave-apps.old directories, and bundle under var/run/splunk/cluster/remote-bundles
   2. Sometimes, stopping CP does not stop child processes and keep updating the bundles and splunkd.log. When running into this state, you need to kill splunkd processes by (killall splunk)
   3. Create Another bundle on Cluster Master
   4. restart Cluster Peer.