Deployment Architecture

We have to remove two search heads from the cluster and add 2 new one

sim_tcr
Communicator

Hello,

We are on splunk search clustering with 4 search heads with version 6.3.3.
We have to remove two search heads from the cluster and add two new brand servers back to the pool.

How can we do this by migrating all user saved items from the old ones to new ones?

Thanks,
Simon Mandy

Tags (1)
0 Karma

jplumsdaine22
Influencer

Is there something preventing you from adding the two new ones and then removing the old ones?

0 Karma

skalliger
SplunkTrust
SplunkTrust

Hi,

all the knowledge object settings which are done via the GUI are replicated amongst the Search Heads (see here what gets replicated by default).
Except the user specific search history, you don't have to migrate anything (and: do you really need the search history on the new ones?).

I would start by adding the 2 new members to the cluster, verify they replicated all settings correctly and then remove the two old SHs from the captain's CLI.

Beware of the captain though. You have to check which server is the captain and if one of those old SHs is the captain, you have to specify another members as the captain (control captaincy).

Skalli

sim_tcr
Communicator

So basically i got to run below on the new member?

splunk init shcluster-config -auth admin:<password> -mgmt_uri https://<newservername>:8089 -replication_port 34567 -replication_factor 4 -conf_deploy_fetch_url https://<ourshcdeployername>:8089 -secret <password> -shcluster_label <ourlabel>

splunk restart



splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089
0 Karma

skalliger
SplunkTrust
SplunkTrust

Almost correct.
Your last statement actually differs if you're sending this command from the new member OR an existing member.
So this:

splunk add shcluster-member -current_member_uri https://<anyexistingmembername>:8089

is wrong. You can only use -current_member_uri from the new Search Head with its own URI.

However, I prefer using the more logically one:

splunk add shcluster-member -new_member_uri <URI>:<management_port>

Sending this command from an existing SH cluster member where URI is the new host.

Skalli

Edit: typo

0 Karma

sim_tcr
Communicator

that worked nicely. user saved items got synced to new search head.

Now to permanently remove the old one, i got to run below from the old search head to be removed?

splunk remove shcluster-member
splunk stop
0 Karma

skalliger
SplunkTrust
SplunkTrust

Yep, just wait 2-3 minutes after removing the member before you stop it. Job done.

0 Karma

sim_tcr
Communicator

so when added the new search head, I did below on the new search head to get it added to index master dashboard (Search Heads tab)
splunk edit cluster-config -mode searchhead -master_uri https://<8089 -secret="">

what would be the command to remove our old search head from the index master dashboard?

0 Karma

skalliger
SplunkTrust
SplunkTrust

What do you mean by "index master dashbaord"?
Are you talking about the Distributed Management Console (DMC)? Does your master also host the DMC? (whose URL looks like this: https://URI:port/en-US/app/splunk_monitoring_console/...)

There are two things you have to do (if you're also hosting the DMC there):
1. Settings > Distributed Search > Search peers > Select "Delet" next to the appropriate SH. This will delete them from the deployer (note that the "master" is the appropriate term for the indexer cluster, whereas "deployer" is the term used for a SH cluster).
2. If your DMC is also hosted there, you can simply rebuild your hosts like this: Open the DMC overview > Settings (in the DMC menu bar) > General Setup > Switch from "Distributed" to "Standalone" and clik apply. After that, click "Distributed" again and apply changes again. The old servers should be gone from the DMC then.

Skalli

0 Karma

sim_tcr
Communicator

No i am not talking about DMC, but indexmaster:8080/en-US/manager/system/clustering?tab=searchheads

0 Karma

skalliger
SplunkTrust
SplunkTrust

They should disappear after some time.

0 Karma

sim_tcr
Communicator

last question,
we came to search head clustering from single search head. So we had copied user items manually from that single search head to clustered search heads.
So will those items also get replicated to newly added server?

0 Karma

skalliger
SplunkTrust
SplunkTrust

Depends where you copied those configurations. Usually, you would copy the configurations to the deployer and let the deployer push the complete bundle to the search heads.

Please read the documentation carefully. Seems like you are still at the beginning of Splunk, so you might want to read about the roles of master, deployer, deployment server and how SHC and indexer clusters work.
See here for migrating from a standalone SH to a SHC.

Skalli

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...