Maintenance is being done in a Splunk cluster from which certain old configurations have to be removed, and new configurations added, since the servers have a new interface of 10GB vs 1gb (previous configuration). Therefore, it is required that the internal communication be done by the new interfaces that have another IP range.
Step 1- assign captain via cli to a new server.
a- Remove all search heads (SH) from the cluster since they will be added with another ip segment.
I think that, in theory, it would work since the Master has the new configurations of the new SH and index. When assigning this new SH, the command should work and automatically Splunk should assign him as captain. It is known that, at least, there must be 3 search heads for this configuration to be valid.
Step 2- configure captain in static mode and avoid that splunk reassigns this mode.
a.1 - When a new SH is added to the cluster, Splunk assigns a new captain at every moment ?, since it was validated that when deleting and adding a new SH, Splunk was synchronized (it took a while), reallocating a new one SH as captain.
Information was searched on the web which was given with this parameter, which prevents a specific S.H from being chosen as captain. This configuration should be stored in the server.conf? What is the parameter that is added to this file?
With the above procedure, Splunk is informed that the removed SH would not be used again, but since the activity could not be performed at the time of returning to the previous configuration, it was presented that the eliminated SH still belonged to a cluster. Therefore, when adding it again, it had to be cleaned and added again.
Any response is appreciated in order to clarify the doubts in these incidents.