We are on splunk search clustering with 4 search heads with version 6.3.3.
We have to remove two search heads from the cluster and add two new brand servers back to the pool.
How can we do this by migrating all user saved items from the old ones to new ones?
all the knowledge object settings which are done via the GUI are replicated amongst the Search Heads (see here what gets replicated by default).
Except the user specific search history, you don't have to migrate anything (and: do you really need the search history on the new ones?).
I would start by adding the 2 new members to the cluster, verify they replicated all settings correctly and then remove the two old SHs from the captain's CLI.
Beware of the captain though. You have to check which server is the captain and if one of those old SHs is the captain, you have to specify another members as the captain (control captaincy).
So basically i got to run below on the new member?
splunk init shcluster-config -auth admin:<password> -mgmt_uri https://<newservername>:8089 -replication_port 34567 -replication_factor 4 -conf_deploy_fetch_url https://<ourshcdeployername>:8089 -secret <password> -shcluster_label <ourlabel> splunk restart splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089
Your last statement actually differs if you're sending this command from the new member OR an existing member.
splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089
is wrong. You can only use -current_member_uri from the new Search Head with its own URI.
However, I prefer using the more logically one:
splunk add shcluster-member -new_member_uri <URI>:<management_port>
Sending this command from an existing SH cluster member where URI is the new host.
that worked nicely. user saved items got synced to new search head.
Now to permanently remove the old one, i got to run below from the old search head to be removed?
splunk remove shcluster-member splunk stop
so when added the new search head, I did below on the new search head to get it added to index master dashboard (Search Heads tab)
splunk edit cluster-config -mode searchhead -master_uri https://<8089 -secret="">
what would be the command to remove our old search head from the index master dashboard?
What do you mean by "index master dashbaord"?
Are you talking about the Distributed Management Console (DMC)? Does your master also host the DMC? (whose URL looks like this: https://URI:port/en-US/app/splunk_monitoring_console/...)
There are two things you have to do (if you're also hosting the DMC there):
1. Settings > Distributed Search > Search peers > Select "Delet" next to the appropriate SH. This will delete them from the deployer (note that the "master" is the appropriate term for the indexer cluster, whereas "deployer" is the term used for a SH cluster).
2. If your DMC is also hosted there, you can simply rebuild your hosts like this: Open the DMC overview > Settings (in the DMC menu bar) > General Setup > Switch from "Distributed" to "Standalone" and clik apply. After that, click "Distributed" again and apply changes again. The old servers should be gone from the DMC then.
we came to search head clustering from single search head. So we had copied user items manually from that single search head to clustered search heads.
So will those items also get replicated to newly added server?
Depends where you copied those configurations. Usually, you would copy the configurations to the deployer and let the deployer push the complete bundle to the search heads.
Please read the documentation carefully. Seems like you are still at the beginning of Splunk, so you might want to read about the roles of master, deployer, deployment server and how SHC and indexer clusters work.
See here for migrating from a standalone SH to a SHC.