Security

Some troubles when add host to SHC

GenRockeR
Explorer

Hi, guys.

I've Splunk Search Head cluster and I want add new member to this cluster. I use documentation from https://docs.splunk.com/Documentation/Splunk/7.2.6/DistSearch/Addaclustermember.
After adding member KVstore doesn't synchronize with clustered KVStore. In log files I finded information about SSL certificate troubles.

2019-04-25T15:53:52.841Z I CONTROL [initandlisten] ** WARNING: No SSL certificate validation can be performed since no CA file has been provided
2019-04-25T15:53:52.841Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

I already generate new certs with splunk commands:

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/auth -n server -l 4096 -p

And add cacert info to server.conf

[kvstore]
disabled = false
sslCAFile = /opt/splunk/etc/auth/cacert.pem

"splunk show kvstore-status" shows short read in log and not synchronized status

host:8191
                     configVersion : -1
                       hostAndPort : host:8191
                     lastHeartbeat : Fri Apr 26 11:39:48 2019
                 lastHeartbeatRecv : ZERO_TIME
              lastHeartbeatRecvSec : 0
                  lastHeartbeatSec : 1556267988.693
                        optimeDate : ZERO_TIME
                     optimeDateSec : 0
                            pingMs : 0
                 replicationStatus : Down
                            uptime : 0

What can I do with this trouble?

0 Karma

rjteh_splunk
Splunk Employee
Splunk Employee

Hi @GenRockeR ,

You can try to clean the local kvstore on the SH that you attempted to add to the SHC by following the steps below.

  1. Stop the search head that has the stale KV store member. (stop splunkd on esh5.prod)
  2. Run the command splunk clean kvstore --local.
  3. Restart the search head. This triggers the initial synchronization from other KV store members.
  4. Run the command splunk show kvstore-status to verify synchronization.

The reference these steps can be found here:

https://docs.splunk.com/Documentation/Splunk/7.2.6/Admin/ResyncKVstore#Resync_stale_KV_store_members

0 Karma

GenRockeR
Explorer

Hi. I already try clean local KVstore, but still have troubles with replication. In mongod.log I found replication errors and short read connection status

0 Karma
Get Updates on the Splunk Community!

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Your Voice Matters! Help Us Shape the New Splunk Lantern Experience

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Building Momentum: Splunk Developer Program at .conf25

At Splunk, developers are at the heart of innovation. That’s why this year at .conf25, we officially launched ...