I would like to delete an index in Splunk using the following command.
splunk remove index
Just wondering where I should run this command? e.g. only on the Cluster Master (CM)? or on the indexers and CM?
I'm aware that data which is pointed to this index will be lost, any other risks I should be aware of?
In a distributed peer/indexer cluster you remove the index from /opt/splunk/etc/master-apps/_cluster/local/indexes.conf OR /opt/splunk/etc/master-apps/YourIndexApp/(local|default)/indexes.conf on the cluster master and then push the cluster configuration bundle from the cluster master.
You should read this first:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Managecommonconfigurations
then this:
http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Configurethepeerindexes
Does this method of deleting indexes only remove the index from searching, but the data in that index is still stored on the servers? is that correct?
Yes that is correct, it leaves the index on the disk still consuming disk space but stops any additional events from being written to the index and stops allowing it to be searched. To remove the files you have to log into each indexer and remove the index under /opt/splunk/var/lib/splunk/ (I believe that's the path, usually get it mixed up but /opt/splunk/var/lib/ is the root dir for sure).
The path the old data will be stored in will depend on the home data path and cold data path settings that were in indexes.conf.