I got the similar issue
04-22-2020 23:08:28.867 +0800 ERROR ClusterMasterPeerHandler - Cannot add peer=xxx.xx.xx.xxx mgmtport=8089 (reason: bucket already added as clustered, peer attempted to add again as standalone. guid=2A40ED04-E90B-4771-BD5A-F523865808B6 bid= ~2~2A40ED04-E90B-4771-BD5A-F523865808B6).
thanks to cluster, My data safe to another peer node and it was searchable.
Stop my index peer which was no behaving "./splunk stop"
Move the whole index dir as _old (mine was in defalut dir - $splunk/var/lib/splunk/)
Start the index again. "./splunk start"
Vola ! error gone and both peer show "up" in index cluster, that the good sign (durability and searchable was still red )
Last, Start the re-balance
There is the feature "data re-balance"
Setting -> index clustering -> edit -> Data rebalance -> Select the index which you moved (I kept threshold 1, and it worked for me) -> start
My index was re-created and data came back on issue peer automatically, all green from UI, availability was 100% and all data was searchable.
-Rana
... View more