If two cluster peers in an indexer cluster are getting close to capacity, how do we stop replicating buckets to these cluster peers?

[sat94541]

We are using Splunk Indexer Clustering and have four Cluster Peers (old) + two Cluster Peers (new) .
We are running close to the storage limit on two of the four cluster peers, so we have added two more new cluster peers.

We have configured the forwarder to stop sending data to the two old cluster peers at capacity, but the cluster master is still replicating data to them.
At this rate, soon we will run out of storage on these two peers.

What are my options? How can I stop replicating buckets to these peers?

[rbal_splunk]

In order to Manage this situation starting Splunk Version 6.3 and above Splunk has added capability to Put Cluster Peer in detention. Once Peer is in detention it will no longer get replicated buckets from other peer in cluster.

The command to put cluster peer in detention is
curl -k -u admin:changeme https://SLAVE:MGMT_PORT/services/cluster/slave/control/control/set_detention_override -d value=true -X POST

Also documented at link --http://docs.splunk.com/Documentation/Splunk/6.3.0/RESTREF/RESTcluster

Only issue is that this setting is not persistent – so if CM/peer is restarted thsi setting is lost setting is lost and we have logged Bug to address this issue . Bug# SPL-109550:[INDX CLUSTER] Make set_detention_override persistent)