topic Re: Issue with kvstore in version 9.4.3 in Splunk Enterprise

Issue with kvstore in version 9.4.3

sushilkar — Tue, 12 Aug 2025 17:30:04 GMT

Having issue with starting kvstore on 9.4.3 on RHEL8.x

Error seen:

ERROR CertStorageProvider [1067727 KVStoreConfigurationThread] - CertificateDatabase is not ready. Cannot sync the certificates stored in CertificateDatabase with the local filesyste

mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)

# sudo ls -l $SPLUNK_HOME/bin/mongod
lrwxrwxrwx. 1 140653 65536 10 May 27 19:01 /opt/splunk/bin/mongod -> mongod-7.0

It is not starting mongo server and hence.

ERROR KVStorageProvider [1067727 KVStoreConfigurationThread] - An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '<mysearchheadhost>:8191']

ldd /opt/splunk/bin/mongod

linux-vdso.so.1 (0x00007ffc12932000)

libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f8c0c38b000)

libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f8c0c16d000)

libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8c0bf18000)

libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f8c0bcc9000)

liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f8c0bab9000)

libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8c0b8a1000)

libcrypto.so.10 => not found

libssl.so.10 => not found

libdl.so.2 => /lib64/libdl.so.2 (0x00007f8c0b69d000)

librt.so.1 => /lib64/librt.so.1 (0x00007f8c0b495000)

libm.so.6 => /lib64/libm.so.6 (0x00007f8c0b113000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8c0aefb000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8c0acdb000)

libc.so.6 => /lib64/libc.so.6 (0x00007f8c0a905000)

/lib64/ld-linux-x86-64.so.2 (0x00007f8c14c87000)

libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f8c0a6de000)

libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f8c0a4c0000)

libssh.so.4 => /lib64/libssh.so.4 (0x00007f8c0a250000)

libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f8c0a03f000)

libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8c09daa000)

libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8c098bf000)

libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8c095d4000)

libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8c093bd000)

libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8c091b9000)

libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8c08fac000)

libz.so.1 => /lib64/libz.so.1 (0x00007f8c08d94000)

libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8c08b6b000)

libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8c0895a000)

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8c08756000)

libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f8c083d5000)

libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f8c081b4000)

libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8c07f89000)

libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f8c07d05000)

I tried setting already the LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH

Re: Issue with kvstore in version 9.4.3

kiran_panchavat — Wed, 13 Aug 2025 02:59:18 GMT

@sushilkarCould you please run the following command and share the output?

/opt/splunk/bin/splunk show kvstore-status --verbose

Also, have you recently upgraded Splunk? If yes, please specify the version you upgraded from and the version you upgraded to.

Re: Issue with kvstore in version 9.4.3

PrewinThomas — Wed, 13 Aug 2025 05:17:52 GMT

@sushilkar

Your main error highlights missing libcrypto.so.10 and libssl.so.10 libraries.

Is this(9.4.3) fresh installation or upgraded from older version? Also RHEL upgrade happened?

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 05:25:34 GMT

It is an upgrade.

There is OS upgrade. to RHEL 8.x

Splunk SH is 3 node cluster. out of 2 are 9.2.6(RHEL7.x) yet to finish upgrade and upgraded one where I see the issue is 9.4.3 on RHEL8.x. SH cluster status is up. But the kvstore isn't coming up, the mongodb process itself is not starting. When I attempted to run manually, it is segfaulting probably lib issue.

RHEL8.x doesn't have older openSSL versions, and with that already fix is there in 9.4.3 I believe.

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 05:27:07 GMT

9.2.6 Run KV store with mongo 4.2

9.4.3 on /RHEL8.x is trying to run on mongodb 7.0

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 05:38:30 GMT

There is base os upgrade as well. New OS is RHEL8.x

and Splunk Version 9.4.3. This is trying to start and mongod.log has the library issues.

Other nodes in the SH cluster is 9.2.6 (on RHEL7.x) in the process of upgrade running kvstore mongo with 4.2 which is showing file.

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 06:40:46 GMT

/opt/splunk/bin/splunk show kvstore-status --verbose

This member:

backupRestoreStatus : Ready

disabled : 0

featureCompatibilityVersion : 4.2

guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB

port : 8191

standalone : 0

status : starting

storageEngine : wiredTiger

versionUpgradeInProgress : 0

Enabled KV store members:

foo1.example.com:8191

backupInProgress : 0

guid : 10354019-RDFE-40DF-9922-6BAF7C3CB00D

hostAndPort : foo1.example.com:8191

foo3.example.com:8191

backupInProgress : 0

guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB

hostAndPort : foo3.example.com:8191

foo2.example.com:8191

backupInProgress : 0

guid : FD6813D8-895D-34DF-23CD-90429FC24229

hostAndPort : foo2.example.com:8191

Mongod is not running:

# ps -ef | grep [m]ongo

# tail -n 3 /opt/splunk/var/log/splunk/mongod.log
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)

This works on RHEL7.x mongodb 4.2

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version
db version v4.2.17-linux-splunk-v4
git version: be089838c55d33b6f6039c4219896ee4a3cd704f
OpenSSL version: OpenSSL 1.0.2zk-fips 3 Sep 2024
allocator: tcmalloc
modules: none
build environment:
distmod: rhel62
distarch: x86_64
target_arch: x86_64

Where as the RHEL8.x one even with LD PATH

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by /opt/splunk/bin/mongod)
Illegal instruction (core dumped)
$ ll /opt/splunk/lib/libcrypto.so.10
lrwxrwxrwx. 1 140653 65536 18 May 27 19:01 /opt/splunk/lib/libcrypto.so.10 -> libcrypto.so.1.0.0

Re: Issue with kvstore in version 9.4.3

thahir — Wed, 13 Aug 2025 07:22:46 GMT

Hi @sushilkar ,

libcrypto.so.10 and libssl.so.10 is missing after the RHEL upgrade that is the actual issue from the logs.

Mongod is not able to find those libraries, that's the cause for the KVstore issue.

Installing the libraries will fix the issue.

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 10:55:42 GMT

Yes that is correct. RHEL8.x has later openssl libraries as default and for splunk and mongodb it need the specific version, so it is shipped along with $SPLUNK_HOME/lib directory with the splunk_enterprse package. Ideally $ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version this should work but in my case:

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 13:27:49 GMT

This looks to be issue with

https://docs.splunk.com/Documentation/Splunk/9.4.2/Installation/AboutupgradingREADTHISFIRST?utm_source=chatgpt.com#Key_points_for_upgrading_to_version_9.4

KV Store server version 7.0 introduces several CPU architecture requirements for the computers that run Splunk Enterprise. Any of these computers must use a CPU that supports Advanced Vector Extensions (AVX), Streaming SIMD

I will close this thread, after testing to a different host.

Re: Issue with kvstore in version 9.4.3

sushilkar — Wed, 13 Aug 2025 13:28:59 GMT

Unfortunately the test host was in old micro architecture with no avx support