Issue with kvstore in version 9.4.3

[sushilkar]

Having issue with starting kvstore on 9.4.3 on RHEL8.x

Error seen:

ERROR CertStorageProvider [1067727 KVStoreConfigurationThread] - CertificateDatabase is not ready. Cannot sync the certificates stored in CertificateDatabase with the local filesyste

mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)

# sudo ls -l $SPLUNK_HOME/bin/mongod
lrwxrwxrwx. 1 140653 65536 10 May 27 19:01 /opt/splunk/bin/mongod -> mongod-7.0

It is not starting mongo server and hence.

ERROR KVStorageProvider [1067727 KVStoreConfigurationThread] - An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '<mysearchheadhost>:8191']

ldd /opt/splunk/bin/mongod

linux-vdso.so.1 (0x00007ffc12932000)

libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f8c0c38b000)

libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f8c0c16d000)

libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8c0bf18000)

libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f8c0bcc9000)

liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f8c0bab9000)

libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8c0b8a1000)

libcrypto.so.10 => not found

libssl.so.10 => not found

libdl.so.2 => /lib64/libdl.so.2 (0x00007f8c0b69d000)

librt.so.1 => /lib64/librt.so.1 (0x00007f8c0b495000)

libm.so.6 => /lib64/libm.so.6 (0x00007f8c0b113000)

libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8c0aefb000)

libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8c0acdb000)

libc.so.6 => /lib64/libc.so.6 (0x00007f8c0a905000)

/lib64/ld-linux-x86-64.so.2 (0x00007f8c14c87000)

libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f8c0a6de000)

libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f8c0a4c0000)

libssh.so.4 => /lib64/libssh.so.4 (0x00007f8c0a250000)

libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f8c0a03f000)

libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8c09daa000)

libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8c098bf000)

libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8c095d4000)

libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8c093bd000)

libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8c091b9000)

libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8c08fac000)

libz.so.1 => /lib64/libz.so.1 (0x00007f8c08d94000)

libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8c08b6b000)

libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8c0895a000)

libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8c08756000)

libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f8c083d5000)

libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f8c081b4000)

libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8c07f89000)

libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f8c07d05000)

I tried setting already the LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH 

[thahir]

Hi @sushilkar ,

libcrypto.so.10 and libssl.so.10 is missing after the RHEL upgrade that is the actual issue from the logs.

Mongod is not able to find those libraries, that's the cause for the KVstore issue.

Installing the libraries will fix the issue.

[sushilkar · Accepted solution]

This looks to be issue with 

https://docs.splunk.com/Documentation/Splunk/9.4.2/Installation/AboutupgradingREADTHISFIRST?utm_sour...

KV Store server version 7.0 introduces several CPU architecture requirements for the computers that run Splunk Enterprise. Any of these computers must use a CPU that supports Advanced Vector Extensions (AVX), Streaming SIMD

I will close this thread, after testing to a different host. 

[sushilkar]

Unfortunately the test host was in old micro architecture with no avx support

[sushilkar]

Yes that is correct. RHEL8.x has later openssl libraries as default and for splunk and mongodb it need the specific version, so it is shipped along with $SPLUNK_HOME/lib directory with the splunk_enterprse package. Ideally $ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version this should work but in my case:

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by /opt/splunk/bin/mongod)
Illegal instruction (core dumped)
$ ll /opt/splunk/lib/libcrypto.so.10
lrwxrwxrwx. 1 140653 65536 18 May 27 19:01 /opt/splunk/lib/libcrypto.so.10 -> libcrypto.so.1.0.0

[PrewinThomas]

@sushilkar 

Your main error highlights missing libcrypto.so.10 and libssl.so.10 libraries.

Is this(9.4.3) fresh installation or upgraded from older version? Also RHEL upgrade happened?

Regards,
Prewin
Splunk Enthusiast | Always happy to help! If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[sushilkar]

It is an upgrade. 

There is OS upgrade. to RHEL 8.x

Splunk SH is 3 node cluster. out of 2 are 9.2.6(RHEL7.x) yet to finish upgrade and upgraded one where I see the issue is 9.4.3 on RHEL8.x. SH cluster status is up. But the kvstore isn't coming up, the mongodb process itself is not starting. When I attempted to run manually, it is segfaulting probably lib issue.

RHEL8.x doesn't have older openSSL versions, and with that already fix is there in 9.4.3 I believe.

[sushilkar]

9.2.6 Run KV store with mongo 4.2

9.4.3 on /RHEL8.x is trying to run on mongodb 7.0

[kiran_panchavat]

@sushilkarCould you please run the following command and share the output?

/opt/splunk/bin/splunk show kvstore-status --verbose 

Also, have you recently upgraded Splunk? If yes, please specify the version you upgraded from and the version you upgraded to.

[sushilkar]

/opt/splunk/bin/splunk show kvstore-status --verbose

 This member:

           backupRestoreStatus : Ready

                      disabled : 0

   featureCompatibilityVersion : 4.2

                          guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB

                          port : 8191

                    standalone : 0

                        status : starting

                 storageEngine : wiredTiger

      versionUpgradeInProgress : 0

 Enabled KV store members:

foo1.example.com:8191

              backupInProgress : 0

                          guid : 10354019-RDFE-40DF-9922-6BAF7C3CB00D

                   hostAndPort : foo1.example.com:8191

foo3.example.com:8191

              backupInProgress : 0

                          guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB

                   hostAndPort : foo3.example.com:8191

foo2.example.com:8191

              backupInProgress : 0

                          guid : FD6813D8-895D-34DF-23CD-90429FC24229

                   hostAndPort : foo2.example.com:8191

Mongod is not running:

# ps -ef | grep [m]ongo

# tail -n 3 /opt/splunk/var/log/splunk/mongod.log
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)

This works on RHEL7.x mongodb 4.2

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version
db version v4.2.17-linux-splunk-v4
git version: be089838c55d33b6f6039c4219896ee4a3cd704f
OpenSSL version: OpenSSL 1.0.2zk-fips 3 Sep 2024
allocator: tcmalloc
modules: none
build environment:
distmod: rhel62
distarch: x86_64
target_arch: x86_64

Where as the RHEL8.x one even with LD PATH

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod)
/opt/splunk/bin/mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by /opt/splunk/bin/mongod)
Illegal instruction (core dumped)
$ ll /opt/splunk/lib/libcrypto.so.10
lrwxrwxrwx. 1 140653 65536 18 May 27 19:01 /opt/splunk/lib/libcrypto.so.10 -> libcrypto.so.1.0.0

[sushilkar]

There is base os upgrade as well. New OS is RHEL8.x

and Splunk Version 9.4.3. This is trying to start and mongod.log has the library issues. 

Other nodes in the SH cluster is 9.2.6 (on RHEL7.x) in the process of upgrade running kvstore mongo with 4.2 which is showing file.