I would like to delete all the data in an index from an indexer cluster without restarting the indexers.
Replication factor= 2 and search factor = 2.
Restarting the indexers causes the cluster master to reassign primaries to available searchable bucket copies. Since my system uses accelerated data models, which are not replicated, all the re-assigned primary buckets need to have their data model re-accelerated. In a large production system, this can take a considerable amount of time.
So you cant delete the indexes, per say, without a restart.
However, there are a few things you could potentially do..
index=deleteme | delete
Delete all the events in the index (this doesnt actually delete, but marks them as deleted.) And this clean the index when time permits.
Maintenance Mode and Delete
Put your cluster into maintenance mode, this will prevent the bucket fixup activities across the cluster. While in maintenance mode, one by one, go through your indexers and clean / delete the index.
After you're done, take the cluster out of maintenance mode. You will still need to apply the cluster bundle to remove the index definition. But that will technically require another rolling restart. However, a rolling restart of the cluster will maintain your SF / RF and users should not notice downtime.
There is no way around the DM issue at this time. Splunk 6.4 should introduce the capability of replicated DM acceleration...