Deployment Architecture

Can you give me some tips with the Re-configuration cluster?

Path Finder

Good morning

Maintenance is being done in a Splunk cluster from which certain old configurations have to be removed, and new configurations added, since the servers have a new interface of 10GB vs 1gb (previous configuration). Therefore, it is required that the internal communication be done by the new interfaces that have another IP range.
 Step 1- assign captain via cli to a new server.
a- Remove all search heads (SH) from the cluster since they will be added with another ip segment.

a.1- ./splunk remove shcluster-member -auth admin: xxxx

(by eliminating the SH's one by one, the moment comes when the captain can not be eliminated)

The question is:

When adding the new SH IP to the cluster, will it be possible to assign the captain immediately?

./splunk init shcluster-config -auth admin: kudaw2011 -mgmt_uri https: //newS.H10GB: 8089 -replication_port 9000 -replication_factor 3 -conf_deploy_fetch_url https: // new-deployer10gb: 8089 -secret secret

./splunk transfer shcluster-captain -mgmt_uri https: //newS.H10GB: 8089 -auth admin: xxx

I think that, in theory, it would work since the Master has the new configurations of the new SH and index. When assigning this new SH, the command should work and automatically Splunk should assign him as captain. It is known that, at least, there must be 3 search heads for this configuration to be valid.

Step  2- configure captain in static mode and avoid that splunk reassigns this mode.
a.1 - When a new SH is added to the cluster, Splunk assigns a new captain at every moment ?, since it was validated that when deleting and adding a new SH, Splunk was synchronized (it took a while), reallocating a new one SH as captain.

Information was searched on the web which was given with this parameter, which prevents a specific S.H from being chosen as captain. This configuration should be stored in the server.conf? What is the parameter that is added to this file?

./splunk edit shcluster-config -mode captain -captain_uri ip-searchead-unwanted: 8089 -election false

Step 3- Eliminate an SH from the cluster in a definitive way
a.1 When the removal of an S.H was carried out, the following procedure was used.

a.1.1- ./splunk remove shcluster-member -auth admin: xxxx
a.1.2- ./splunk stop (wait around 2 minutes for synchronization)
The incident that was presented was that when trying to undo the changes and re-add the deleted S.H, splunk reported that this S.H already belonged to a cluster.

a.1.3- ./splunk init shcluster-config -auth admin: kudaw2011 -mgmt_uri https: // sh-old: 8089 -replication_port 9000 -replication_factor 3 -conf_deploy_fetch_url https: // deployer-old: 8089 -secret secret
a.1.4- When not being able to add the S.H, the following was done.

a.1.4.1 - ./splunk stop
a.1.4.2 - ./splunk clear all
a.1.4.3 - ./splunk start
a.1.4.4 - ./splunk init shcluster-config -auth admin: kudaw2011 -mgmt_uri https: // sh-old: 8089 -replication_port 9000 -replication_factor 3 -conf_deploy_fetch_url https: // deployer-old: 8089 -secret secret
a.1.4.5 - ./splunk restart
a.1.4.6 - ./splunk add shcluster-member -new_member_uri https: // sh-old: 8089 -auth admin: xxx
a.1.4.7 - ./splunk show shcluster-status -auth admin: xxx
With the above procedure, Splunk is informed that the removed SH would not be used again, but since the activity could not be performed at the time of returning to the previous configuration, it was presented that the eliminated SH still belonged to a cluster. Therefore, when adding it again, it had to be cleaned and added again.

Any response is appreciated in order to clarify the doubts in these incidents.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!