On the DMC of my cluster master, I've added a few new search peers (in Set Up) and deleted the old ones, I also gave them the correct server roles. However, when go to the overview, it doesn't display the new ones, it displays the old ones. When I go into the tab "instances" I see one of the old instances that I've already deleted and I see one of the new instances I've added, but it still says it's a deployment server, even though I have saved its new role! And the new instance is "unreachable" (no information about the machine), but I can go to the Splunk Web of that instance and it's definitely up. In the search peer page, it also shows that all instances are "up" and does recognize the instance (license signature etc).
Restarting Splunk on the cluster master did not work.
What am I missing here? Adding the new instances as search peers isn't enough?
Thanks in advance.
Are you sure you have network communication functioning on tcp-8089 (splunk default mgmt) between the DMC and the new peers you are adding?
Are you forwarding all of your splunk internal logs from your new search heads to your indexers? Maybe look to see if there are any errors or warnings in the splunkd log in the DistributedPeerManager component on your DMC search head.
(sorry internet was acting up and couldn't finish my first comment)
On to your second comment.. No the search head is not forwarding internal data to the indexers atm. But the old one wasn't doing that either and that one was showing up fine in the cluster master DMC.
I just logged into the new search head and checked the DMC, there were no general messages and also no messages/warnings in DMC.
Thanks for your answer. Port 8089 is turned on for all instances indeed. It's strange, because one of the new instances is a search head. When I go to index clustering it recognizes it as a search head, says it's up and shows all the information.
You will definitely see any search heads configured as cluster clients in your Cluster Manager View. Did you go to settings > distributed search > search peers and add the new search heads there as search peers and remove the old ones?
Yes, I added the search peers on settings > distributed search > search peers. I deleted the old ones there and I added the new ones. One is a new search head and the other one is a heavy forwarder. In the beginning I used to assign the server role as Deployment server but I have changed it and even removed it entirely and added it as a new search peer. The new ones are "up" and "succesful". Then I go to the DMC of the same cluster master. I look at Set Up. I see the 2 new (remote) instances and have changed and saved their server roles. When I click on them, they show all the info about the machines, nothing wrong. They are "enabled" and "configured".
Now, when I go to the Instances page of the DMC, I don't see the new search head, I see the old one which is obviously "unreachable" (since I've deleted and stopped that Linux machine). As for the heavy forwarder: it's also "unreachable" and the role is deployment server! This is strange since I've deleted both search peers..
The same for the overview, I see the old instances but not the new ones.
Sorry just saw I accidentally wrote in the original post I added them in Set Up instead of Distributed search > search peers. Sorry for the confusion.