I am unable to remove search peers from the Distributed Management Console. When I try to remove it from Splunk Web, i get below error:-
Error occurred attempting to remove XXX.XXX.XXX.XX:8089(intentionally masked): Cannot remove peer=https://XXX.XXX.XXX.XX:8089.
This peer is a part of a search head cluster. I have already removed the cluster master from the search peer list. I also tried removing it from splunk_home/etc/system/local/distsearch.conf.
Tried removing using CLI command
splunk remove search-server -auth admin:password XXX.XXX.XXX.XX:8089
but it gives same error and peer persist in the search peer list.
Please let me know how I can remove all search peers which are part of the cluster.
You're mixing up a few terms here. A search peer is an indexer. An indexer is not part of a search head cluster. So, do you want to remove a search peer from your indexer cluster or do you want to remove a search head cluster member?
Depending on your answer, the commands are quite different.
For a search head, you can either use
splunk remove shcluster-member
on your search head (not allowed if it is a captain) or
splunk remove shcluster-member -mgmt_uri <URI>:<management_port>
If it is an indexer, you have to stop it first and then use a command from the master:
splunk remove cluster-peers -peers <guid>
I am able to remove it from the DMC peer list by removing cluster masters from /system/local/server.conf.