×

Join the Conversation

Without signing in, you're just watching from the sidelines. Sign in or Register to connect, share, and be part of the Splunk Community.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question
sushilkar
Explorer
Member since: ‎08-12-2025
‎08-13-2025
Community Statistics
Posts 10
Solutions 1
Karma Given 0
Karma Received 0
Member Since ‎08-12-2025
Activity Feed
View All

Re: KVstore error output after splunk enterprise ...

by in Splunk Enterprise
‎08-13-2025 10:30 PM
‎08-13-2025 10:30 PM
The logs that you see are not harmful, treat it like no version info added library .so file in /opt/lib/splunk its more of a warning. Check if you are able to run mongod --version. (https://community.splunk.com/t5/forums/replypage/board-id/core-splunk-enterprise/message-id/22866)   Check mongod --version you are able to run, see this post.  https://community.splunk.com/t5/Splunk-Enterprise/Issue-with-kvstore-in-version-9-4-3/m-p/751436#M22845   Sa ... View more

Re: KVstore error output after splunk enterprise ...

by in Splunk Enterprise
‎08-13-2025 10:25 PM
‎08-13-2025 10:25 PM
mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod:/splunk/lib/libssl.so.10: no version information available (required by mongod)   isn't a harmful error that you should worry about. Splunk ships its own libcrypto.so.10 that doesn’t have the GNU symbol version sections.The dynamic loader warns you, but execution usually continues.  You could check if the mongod is seeing that lib from /opt/splunk/lib overridden path: Simply that error you can ignore as mentioned above.   If loading issue is there: you will see the lib missing $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH ldd $SPLUNK_HOME/bin/mongod    moreover $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version  OR splunk cmd mongod -version this works for you means no issues with KVstore 7.0   Do you see the process mongod not started by SH? pgrep -f mongod , can see if its running  also check the kvstore-status  /opt/splunk/bin/splunk show kvstore-status     ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-13-2025 06:28 AM
‎08-13-2025 06:28 AM
Unfortunately the test host was in old micro architecture with no avx support ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-13-2025 06:27 AM
‎08-13-2025 06:27 AM
This looks to be issue with  https://docs.splunk.com/Documentation/Splunk/9.4.2/Installation/AboutupgradingREADTHISFIRST?utm_source=chatgpt.com#Key_points_for_upgrading_to_version_9.4 KV Store server version 7.0 introduces several CPU architecture requirements for the computers that run Splunk Enterprise. Any of these computers must use a CPU that supports Advanced Vector Extensions (AVX), Streaming SIMD I will close this thread, after testing to a different host.  ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-13-2025 03:55 AM
‎08-13-2025 03:55 AM
Yes that is correct. RHEL8.x has later openssl libraries as default and for splunk and mongodb it need the specific version, so it is shipped along with $SPLUNK_HOME/lib directory with the splunk_enterprse package. Ideally $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version this should work but in my case: $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by /opt/splunk/bin/mongod) Illegal instruction (core dumped) $ ll /opt/splunk/lib/libcrypto.so.10 lrwxrwxrwx. 1 140653 65536 18 May 27 19:01 /opt/splunk/lib/libcrypto.so.10 -> libcrypto.so.1.0.0 ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-12-2025 11:40 PM
‎08-12-2025 11:40 PM
/opt/splunk/bin/splunk show kvstore-status --verbose    This member:            backupRestoreStatus : Ready                       disabled : 0    featureCompatibilityVersion : 4.2                           guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB                           port : 8191                     standalone : 0                         status : starting                  storageEngine : wiredTiger       versionUpgradeInProgress : 0    Enabled KV store members: foo1.example.com:8191               backupInProgress : 0                           guid : 10354019-RDFE-40DF-9922-6BAF7C3CB00D                    hostAndPort : foo1.example.com:8191 foo3.example.com:8191               backupInProgress : 0                           guid : 41CAFB1B-XDFT-4232-98EB-8C83F2B7FEBB                    hostAndPort : foo3.example.com:8191 foo2.example.com:8191               backupInProgress : 0                           guid : FD6813D8-895D-34DF-23CD-90429FC24229                    hostAndPort : foo2.example.com:8191   Mongod is not running: # ps -ef | grep [m]ongo   # tail -n 3 /opt/splunk/var/log/splunk/mongod.log mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)   This works on RHEL7.x mongodb 4.2 $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version db version v4.2.17-linux-splunk-v4 git version: be089838c55d33b6f6039c4219896ee4a3cd704f OpenSSL version: OpenSSL 1.0.2zk-fips 3 Sep 2024 allocator: tcmalloc modules: none build environment: distmod: rhel62 distarch: x86_64 target_arch: x86_64   Where as the RHEL8.x one even with LD PATH $ SPLUNK_HOME=/opt/splunk $ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by /opt/splunk/bin/mongod) /opt/splunk/bin/mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by /opt/splunk/bin/mongod) Illegal instruction (core dumped) $ ll /opt/splunk/lib/libcrypto.so.10 lrwxrwxrwx. 1 140653 65536 18 May 27 19:01 /opt/splunk/lib/libcrypto.so.10 -> libcrypto.so.1.0.0 ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-12-2025 10:38 PM
‎08-12-2025 10:38 PM
There is base os upgrade as well. New OS is RHEL8.x and Splunk Version 9.4.3. This is trying to start and mongod.log has the library issues.  Other nodes in the SH cluster is 9.2.6 (on RHEL7.x) in the process of upgrade running kvstore mongo with 4.2 which is showing file. ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-12-2025 10:27 PM
‎08-12-2025 10:27 PM
9.2.6 Run KV store with mongo 4.2 9.4.3 on /RHEL8.x is trying to run on mongodb 7.0 ... View more

Re: Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-12-2025 10:25 PM
‎08-12-2025 10:25 PM
It is an upgrade.    There is OS upgrade. to RHEL 8.x Splunk SH is 3 node cluster. out of 2 are 9.2.6(RHEL7.x) yet to finish upgrade and upgraded one where I see the issue is 9.4.3 on RHEL8.x. SH cluster status is up. But the kvstore isn't coming up, the mongodb process itself is not starting. When I attempted to run manually, it is segfaulting probably lib issue.   RHEL8.x doesn't have older openSSL versions, and with that already fix is there in 9.4.3 I believe. ... View more

Issue with kvstore in version 9.4.3

by in Splunk Enterprise
‎08-12-2025 10:30 AM
‎08-12-2025 10:30 AM
Having issue with starting kvstore on 9.4.3 on RHEL8.x   Error seen: ERROR CertStorageProvider [1067727 KVStoreConfigurationThread] - CertificateDatabase is not ready. Cannot sync the certificates stored in CertificateDatabase with the local filesyste   mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod) mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)   # sudo ls -l $SPLUNK_HOME/bin/mongod lrwxrwxrwx. 1 140653 65536 10 May 27 19:01 /opt/splunk/bin/mongod -> mongod-7.0   It is not starting mongo server and hence. ERROR KVStorageProvider [1067727 KVStoreConfigurationThread] - An error occurred during the last operation ('getServerVersion', domain: '15', code: '13053'): No suitable servers found (`serverSelectionTryOnce` set): [connection closed calling hello on '<mysearchheadhost>:8191']   ldd /opt/splunk/bin/mongod linux-vdso.so.1 (0x00007ffc12932000) libcurl.so.4 => /lib64/libcurl.so.4 (0x00007f8c0c38b000) libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007f8c0c16d000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8c0bf18000) libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f8c0bcc9000) liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f8c0bab9000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8c0b8a1000) libcrypto.so.10 => not found libssl.so.10 => not found libdl.so.2 => /lib64/libdl.so.2 (0x00007f8c0b69d000) librt.so.1 => /lib64/librt.so.1 (0x00007f8c0b495000) libm.so.6 => /lib64/libm.so.6 (0x00007f8c0b113000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8c0aefb000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8c0acdb000) libc.so.6 => /lib64/libc.so.6 (0x00007f8c0a905000) /lib64/ld-linux-x86-64.so.2 (0x00007f8c14c87000) libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007f8c0a6de000) libidn2.so.0 => /lib64/libidn2.so.0 (0x00007f8c0a4c0000) libssh.so.4 => /lib64/libssh.so.4 (0x00007f8c0a250000) libpsl.so.5 => /lib64/libpsl.so.5 (0x00007f8c0a03f000) libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f8c09daa000) libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8c098bf000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8c095d4000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8c093bd000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8c091b9000) libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f8c08fac000) libz.so.1 => /lib64/libz.so.1 (0x00007f8c08d94000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f8c08b6b000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8c0895a000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8c08756000) libunistring.so.2 => /lib64/libunistring.so.2 (0x00007f8c083d5000) libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f8c081b4000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8c07f89000) libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f8c07d05000)   I tried setting already the LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH      ... View more
Labels
Contact Me
Online Status
Offline
Date Last Visited
‎08-13-2025 10:14 PM