Adding an additional Index Peer to an Indexer Cluster

[MakszimM]

Hi Guys,

We are about to expand an Indexing Cluster from 2 to 3 with an additional Index peer.

The process is pretty clear.

Up until the point, where I have to migrate/clone or fill out the server.conf or web.conf files along with the splunk secret.

The other two Indexer servers, upon inspecting the files, have different pem,key and crt files.
I thought these have to be identical?
Do i need to generate a third one, that is different as well, or sync all 3 so that they are identical.
Any guidance is appreciated !
Thank you

[MakszimM]

Hi @sainag_splunk 

Thanks for the detailed reply!
The steps for setting it up is clear so far.
Do I have to copy out the cluster key, splunk secret, if so, how can I do that?

Or can I simply  do a 

$SPLUNK_HOME/bin/splunk rotate splunk-secret --accept-license --answer-yes --no-prompt --seed-passwd <admin_password>

Thank you!

[thahir]

@MakszimM ,

Please follow the steps which is provided by @sainag_splunk to add the additional indexers peers to the IDX cluster.

To answer your question,

Copy the splunk.secret from the old peer to the new per and place it under $splunk_home/etc/auth, before that take the backup of the secret file.

execute the below command on the new peer with your client secret key.

 $SPLUNK_HOME/bin/splunk edit cluster-config -mode peer -manager_uri https://<manager_IP>:8089 -secret <original_plain_text_pass4SymmKey>

[sainag_splunk]

@MakszimM 

If you have the original plain text pass4SymmKey, you can simply run:

  $SPLUNK_HOME/bin/splunk edit cluster-config -mode peer -manager_uri https://<manager_IP>:8089 -secret <original_plain_text_pass4SymmKey>

  The -secret parameter takes the original plain text key, not the hashed version from the config files. Splunk will automatically hash it during the configuration process.


Forgot pass4SymmKey? https://community.splunk.com/t5/Security/Forgot-Pass4symmKey/m-p/378993

If this Helps, Please Upvote!

[sainag_splunk]

Hi @MakszimM,

Regarding the SSL certificates - the existing indexers having different certificate files is normal and acceptable. Each indexer can maintain its own SSL certificates. For your new 3rd indexer, you can either generate new unique certificates or copy existing ones based on your security and operational requirements.

For adding the 3rd indexer, follow these steps:

On the New Indexer Peer:

  • Configure the new server as a peer node to the existing indexer cluster:

  $SPLUNK_HOME/bin/splunk edit cluster-config -mode peer -manager_uri https://<cluster_manager_IP>:8089 -replication_port 9887 -secret  <your_cluster_key>

  • Restart Splunk for changes to take effect:

  $SPLUNK_HOME/bin/splunk restart

  • Verify the peer appears in the cluster manager dashboard

  On the Cluster Manager:

  • Update the Replication Factor and Search Factor (RF=3, SF=2 for 2 peer failure tolerance)

  • Apply the cluster bundle to distribute configuration changes

  • Rebalance the cluster data to distribute existing buckets to the new peer:

  $SPLUNK_HOME/bin/splunk rebalance cluster-data

  • Monitor the rebalancing process in the Cluster Manager dashboard

  • Verify bucket distribution and remove any excess buckets if present

  Reference Links:

https://help.splunk.com/en/splunk-enterprise/administer/manage-indexers-and-indexer-clusters/9.0/man...

https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Rebalancethecluster

The key point is ensuring the cluster secret matches across all peers, while SSL certificates can remain unique per indexer.