topic Changing peer IP address ? in Deployment Architecture

Changing peer IP address ?

fde — Thu, 02 May 2024 20:12:46 GMT

Hello,

I've got a cluster with 2 peers, 1 seach head and 1 CM. All of them with a single network.

Due to network change, the server are going to have an additionnal card with a new network address.

I'll like to know if it's possible to swap the IP address used for réplication between peer member and SH communication while keeping the old one for forwarder communication

Initialy:

peer 1 => 10.254.x.1, peer 2 => 10.254.x.2

After changes :

Peer 1 => forwarder communication, 10.254.x.1, réplication/SH comm=> 10.254.y.1

Peer 2 => forwarder communication, 10.254.x.2, réplication/SH comm=> 10.254.y.2

I've try to use register_replication_address and register_search_address parameter in server.conf with the new address 10.254.y. but the peer and the CM, complain of duplicate guid/member.

Do you have any advice on how to do this, if it's possible ?

Thanks

Frédéric

Re: Changing peer IP address ?

gcusello — Fri, 03 May 2024 06:25:56 GMT

Hi @fde ,

I'm not sure that's possible to use two IP addresses for each server: in Splunk every hostname has one IP address.

Ciao.

Giuseppe

Re: Changing peer IP address ?

fde — Fri, 03 May 2024 06:38:48 GMT

Hi,

I've seen in server.conf specification under clustering stanza, different parameter (https://docs.splunk.com/Documentation/Splunk/9.0.7/Admin/Serverconf#High_availability_clustering_configuration)

register_search_address: "This is the address that advertises the peer to search heads. This is useful in the cases where a splunk host machine has multiple interfaces and only one of them can be reached by another splunkd instance."

register_forwarder_address: "This is the address on which a peer is available for accepting data from forwarder.This is useful in the cases where a splunk host machine has multiple interfaces and only one of them can be reached by another splunkd instance."

register_replication_address: "This is the address on which a peer is available for accepting replication data. This is useful in the cases where a peer host machine has multiple interfaces and only one of them can be reached by another splunkd instance."

So It's seems to be possible to use multiple IP, but I'm wordering how to convert peer with single IP to peer with multiple IP using this parameter.

Frédéric

Re: Changing peer IP address ?

fde — Fri, 03 May 2024 12:07:53 GMT

Here is the message logged on peer side, when during starting after adding register_replication_address

05-02-2024 11:26:38.046 +0200 WARN CMSlave [1355571 indexerPipe] - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json manager=CM:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=NEW-IP mgmtport=8089 (reason: Peer with guid=F00C07FD-F8A6-4C15-91D3-8A6CDCF28C96 is already registered and UP). [ event=addPeer status=retrying AddPeerRequest: { active_bundle_id=054CB061CFCD038A4B98FCDB01CE3F2F add_type=Initial-Add base_generation_id=0 batch_serialno=1 batch_size=26 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 guid=F00C07FD-F8A6-4C15-91D3-8A6CDCF28C96 last_complete_generation_id=0 latest_bundle_id=054CB061CFCD038A4B98FCDB01CE3F2F mgmt_port=8089 register_forwarder_address= register_replication_address=NEW-IP register_search_address=NEW-IP replication_port=8087 replication_use_ssl=0 replications= server_name=PEER-NAME site=site1 splunk_version=9.0.7 splunkd_build_number=b985591d12fd status=Up } Batch 1/26 ].

Re: Changing peer IP address ?

fde — Mon, 13 May 2024 14:30:44 GMT

Hi,

My problem is solved.

With support guidance, I change IP address used for replication.

Here is the steps used to:

1°) stop indexer

2°) change etc/system/local/server.conf and add register_replication_address parameter with the new IP

3°) rename etc/instance.conf to etc/instance.conf.bkp

4°) restart indexer

CM = Index clustering after few seconds => OK
MC reconfigured the Indexer => OK
CM and SH = Distributed Search => 2 entry for the same instance name with 2 different Peer URI, the 2 entry appeared down and Sick

5°) I deleted one entry, the one with the old Peer URI => After 5min and a refresh, no more entry Down and Sick.

And now, all seems to be ok. IP address for peer URI is now showing the new IP address.

Re: Changing peer IP address ?

gcusello — Mon, 13 May 2024 14:37:31 GMT

Hi @fde ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉