Some troubles when add host to SHC

[GenRockeR]

Hi, guys.

I've Splunk Search Head cluster and I want add new member to this cluster. I use documentation from https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Addaclustermember.
After adding member KVstore doesn't synchronize with clustered KVStore. In log files I finded information about SSL certificate troubles.

2019-04-25T15:53:52.841Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
2019-04-25T15:53:52.841Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

I already generate new certs with splunk commands:

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server -l 4096 -p

And add cacert info to server.conf

[kvstore]
disabled = false
sslCAFile = /opt/splunk/etc/auth/cacert.pem

"splunk show kvstore-status" shows short read in log and not synchronized status

host:8191
                     configVersion : -1
                       hostAndPort : host:8191
                     lastHeartbeat : Fri Apr 26 11:39:48 2019
                 lastHeartbeatRecv : ZERO_TIME
              lastHeartbeatRecvSec : 0
                  lastHeartbeatSec : 1556267988.693
                        optimeDate : ZERO_TIME
                     optimeDateSec : 0
                            pingMs : 0
                 replicationStatus : Down
                            uptime : 0

What can I do with this trouble?

[rjteh_splunk]

Hi @GenRockeR ,

You can try to clean the local kvstore on the SH that you attempted to add to the SHC by following the steps below.

1. Stop the search head that has the stale KV store member. (stop splunkd on esh5.prod)
2. Run the command splunk clean kvstore --local.
3. Restart the search head. This triggers the initial synchronization from other KV store members.
4. Run the command splunk show kvstore-status to verify synchronization.

The reference these steps can be found here:

https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/ResyncKVstore#Resync_stale_KV_store_members

[GenRockeR]

Hi. I already try clean local KVstore, but still have troubles with replication. In mongod.log I found replication errors and short read connection status