New indexer installation for Cluster Master

[rayar]

I have installed a new Indexer but I am getting the below error 

looks like the data is copied  but also I don't see the server in  Indexer Clustering: Master Node

2 questions 

1. how to add the new indexer to the list 

2. how to resolve the error message 

02-21-2022 17:52:56.508 +0200 INFO CMSlave - event=addPeer status=failure shutdown=false request: AddPeerRequest: { _id= active_bundle_id=EE37C1F78B2D04FFE51AD60A72882ADB add_type=Initial-Add base_generation_id=0 batch_serialno=1 batch_size=1 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=EE37C1F78B2D04FFE51AD60A72882ADB mgmt_port=8089 name=243C06C6-E196-4E2D-A990-5AD71B271ED5 register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name=ilissplidx11 site=default splunk_version=7.3.4 splunkd_build_number=13e97039fb65 status=Up }
02-21-2022 17:52:56.508 +0200 ERROR CMSlave - event=addPeer start over and retry after sleep 100ms reason= addType=Initial-Add Batch SN=1/1 failed. add_peer_network_ms=3
02-21-2022 17:52:56.608 +0200 INFO CMSlave - event=addPeer Batch=1/1
02-21-2022 17:52:56.611 +0200 WARN CMSlave - Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=illinissplnkmaster:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line="Internal Server Error" socket_error="No error" remote_error=Cannot add peer=10.232.208.35 mgmtport=8089 (reason: http client error=No route to host, while trying to reach https://10.232.208.35:8089/services/cluster/config). [ event=addPeer status=retrying AddPeerRequest: { _id= active_bundle_id=EE37C1F78B2D04FFE51AD60A72882ADB add_type=Initial-Add base_generation_id=0 batch_serialno=1 batch_size=1 forwarderdata_rcv_port=9997 forwarderdata_use_ssl=0 last_complete_generation_id=0 latest_bundle_id=EE37C1F78B2D04FFE51AD60A72882ADB mgmt_port=8089 name=243C06C6-E196-4E2D-A990-5AD71B271ED5 register_forwarder_address= register_replication_address= register_search_address= replication_port=8080 replication_use_ssl=0 replications= server_name=ilissplidx11 site=default splunk_version=7.3.4 splunkd_build_number=13e97039fb65 status=Up } ].

[PickleRick]

The "no route to host" error suggests something wrong with network connectivity. Might be indeed missing routing but can also be firewall or selinux blocking connection.

BTW, you're using 7.3.4 - it's quite an old version. You should really consider an upgrade.

[rayar]

we are now working to plan the upgrade

I see that the Cluster Master is not able to telnet the new indexer 

will check 

[splunk@ilissplmstr01 etc]$ /bin/telnet ilissplidx11 9997
Trying 10.232.208.35...
telnet: connect to address 10.232.208.35: No route to host
[splunk@ilissplmstr01 etc]$ /bin/telnet ilissplidx10 9997
Trying 10.232.209.131...
Connected to ilissplidx10.

but the question is why I am able to see the index buckets created on the ilissplidx11  FS  ?
Escape character is '^]'.

[PickleRick]

We can't tell you without knowing how your network and hosts are configured. I notice that your indexers' addreses differ in third octet so either you use quite "wide" addressing or they are in different subnets.

If your search-heads see all indexers, you will see data from all indexers. It's just that the clustering functionality between indexers will not work (most importantly - data replication).

[SinghK]

Check this link to enable peer node via cli.

https://docs.splunk.com/Documentation/Splunk/8.2.4/Indexer/ConfigurepeerswithCLI

[rayar]

also how it can be done via the GUI  ?

[rayar]

I have updated the server details under /opt/splunk/etc/system/local/distsearch.conf

[distributedSearch:dmc_group_indexer]

[distributedSearch:dmc_indexerclustergroup_Information_Security]

where else the CLI will update  ?