topic Re: How to add a new search head to an existing SHC? in Deployment Architecture

How to add a new search head to an existing SHC?

ram254481493 — Fri, 21 Jun 2019 14:39:08 GMT

Hello , we are planning to add a new search head to our existing search head cluster.
What are the steps I need to follow to properly to add a new member to our existing search head cluster?
We are pushing configs through deployer, so what is a command that I can use to update the new member with existing deployer and other search head settings?

Re: How to add a new search head to an existing SHC?

harsmarvania57 — Fri, 21 Jun 2019 15:05:58 GMT

Hi,

To add new SH in existing SHC, please follow guide https://docs.splunk.com/Documentation/Splunk/7.3.0/DistSearch/Addaclustermember#Add_the_instance.

Re: How to add a new search head to an existing SHC?

amitm05 — Fri, 21 Jun 2019 15:15:02 GMT

You need to initialize and then Add this instance to the cluster. This can be done by running the following commands on your new instance

To initialize, run the below command and restart-

splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key> -shcluster_label <label>  
splunk restart

To Add instance to Cluster, run this command -

splunk add shcluster-member -current_member_uri <URI>:<management_port>

Let me know if this helps

Re: How to add a new search head to an existing SHC?

ram254481493 — Fri, 21 Jun 2019 15:15:27 GMT

Thanks i found the commands to update the configs here for SHC settings but didnt find the resync command to update the deployer configs .

Re: How to add a new search head to an existing SHC?

harsmarvania57 — Fri, 21 Jun 2019 15:17:39 GMT

You do not require to resync, when you add new SH to SHC it will automatically get configuration bundle from Deployer and then it will contact Captain & download replicated config tarball.

Re: How to add a new search head to an existing SHC?

rafiq_rehman — Thu, 22 May 2025 22:42:56 GMT

I followed these steps to add new box to existing SHC, everything looks fine on the SHC side but kvstore status has been 'status : starting' since, and it's been over an hour. Is this normal or I missed something?

Re: How to add a new search head to an existing SHC?

isoutamo — Thu, 22 May 2025 22:46:29 GMT

No it’s not normal. Usually it should be ready quite quickly. Anything in your internal logs?

Re: How to add a new search head to an existing SHC?

rafiq_rehman — Fri, 23 May 2025 01:15:35 GMT

See these events in splunkd:

05-22-2025 21:07:58.608 -0400 ERROR KVStoreAdminHandler [1848035 TcpChannelThread] - An error occurred.
05-22-2025 21:07:36.668 -0400 ERROR KVStoreIntrospection [1848033 TcpChannelThread] - failed to get introspection data
05-22-2025 21:07:19.587 -0400 WARN KVStoreConfigurationProvider [1845927 MainThread] - Action scheduled, but event loop is not ready yet

Tried cleaning up kvstore by running "splunk clean kvstore --local --answer-yes" but that didn't change anything, status is still stuck in starting.

Re: How to add a new search head to an existing SHC?

PrewinThomas — Fri, 23 May 2025 04:46:33 GMT

@rafiq_rehman
Most probably no captain or kvstore port blocked on this new member.

the KVStore cannot reach "Ready" unless a captain is elected and cluster coordination is healthy. If there’s no captain or communication is broken, KVStore remains in "starting"

or kvstore default port 8191 might be blocked. If this port is blocked/network issues, KVStore cannot synchronize and will not start

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a kudos. Thanks!

Re: How to add a new search head to an existing SHC?

isoutamo — Fri, 23 May 2025 04:53:14 GMT

Anything on mongodb.log?
Normally you shouldn’t do any “fixing” activities before you know what and why it’s broken! Those fix could get the situation even worse!

Re: How to add a new search head to an existing SHC?

rafiq_rehman — Thu, 29 May 2025 03:29:18 GMT

It's the existing SHC, just trying to add new member. Captain is working and 8191 port is also opened. This is what I see in mongod log:

2025-05-29T03:22:15.751Z I CONTROL [LogicalSessionCacheReap] Failed to reap transaction table: NotYetInitialized: Replication has not yet been configured 2025-05-29T03:22:15.747Z I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured

Re: How to add a new search head to an existing SHC?

rafiq_rehman — Thu, 29 May 2025 03:31:49 GMT

My bad 🙂

I removed Splunk and just rebuilt it fresh. Still same issue, kvstore has been in starting status. This is what I see in mongod.log:

Re: How to add a new search head to an existing SHC?

PrewinThomas — Thu, 29 May 2025 03:55:28 GMT

@rafiq_rehman
These errors mean the new member’s KV Store is still in standalone mode.

Initialize SHC Config on the New Member

splunk init shcluster-config \
-auth <admin_user>:<admin_pass> \
-mgmt_uri https://<new_member_host>:8089 \
-replication_port <replication_port> \
-replication_factor <factor> \
-conf_deploy_fetch_url https://<deployer_host>:8089 \
-secret <pass4SymmKey> \
-shcluster_label <label>

Add the New Member to the Cluster
splunk add shcluster-member -current_member_uri https://<existing_member_or_captain>:8089

Confirm pass4SymmKey and mgmt_uri in $SPLUNK_HOME/etc/system/local/server.conf

Re: How to add a new search head to an existing SHC?

isoutamo — Thu, 29 May 2025 15:34:58 GMT

You are adding it as a new member like @PrewinThomas shows?

And you are using correct and same format for host names than you are used for those other nodes like FQDN, short names or IPs?

Re: How to add a new search head to an existing SHC?

PickleRick — Thu, 29 May 2025 19:15:01 GMT

Are you sure you're adding the same Splunk version and kvstore version (and engine) instance as the rest of the SHC?