topic KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1 in Splunk Enterprise

KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

swamybhatta — Fri, 04 Apr 2025 20:47:46 GMT

./splunk cmd mongod -version

mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod:/splunk/lib/libssl.so.10: no version information available (required by mongod)
db version v7.0.14
Build Info: {
"version": "7.0.14",
"gitVersion": "ce59cfc6a3c5e5c067dca0d30697edd68d4f5188",
"openSSLVersion": "OpenSSL 1.0.2zk-fips 3 Sep 2024",
"modules": [
"enterprise"
],
"allocator": "tcmalloc",
"environment": {
"distmod": "rhel70",
"distarch": "x86_64",
"target_arch": "x86_64"
}
}

why am i getting mongo db error

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

livehybrid — Fri, 04 Apr 2025 21:35:13 GMT

Hi @swamybhatta

The mongo command isnt intended to be run directly from the CLI - The error you are seeing is likely due to the missing ENV variables such as LD_LIBRARY_PATH.

Does mongo give an error when the processes is created by Splunk as part of the service's boot sequence?

🌟 Did this answer help you? If so, please consider:

Adding kudos to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

GSNRMUVW — Wed, 13 Aug 2025 13:44:48 GMT

In the Splunk UI appears an error...

KV Store process terminated abnormally (exit code 4, status PID 55313 killed by signal 4: Illegal instruction). See mongod.log and splunkd.log for details.

KV Store changed status to failed. KVStore process terminated..

Failed to start KV Store process. See mongod.log and splunkd.log for details.

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

tej57 — Wed, 13 Aug 2025 13:48:15 GMT

Hey @GSNRMUVW,

Did you check the mongod.log? It would indicate some message as an error for terminating the KVStore process. Can you paste the last few lines from the log and mask the sensitive data?

Thanks,
Tejas.

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

GSNRMUVW — Wed, 13 Aug 2025 13:54:37 GMT

splunk@debian:~$ tail -n50 /opt/splunk/var/log/splunk/mongod.log
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libcrypto.so.10: no version information available (required by mongod)
mongod: /opt/splunk/lib/libssl.so.10: no version information available (required by mongod)

In my case, the problem exists since some kernel updates like debian 12.11 and debian 13.0...

https://lists.debian.org/debian-testing/2025/07/msg00035.html

link-systemctl-shared=true/false

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

sushilkar — Thu, 14 Aug 2025 05:25:55 GMT

isn't a harmful error that you should worry about.

Splunk ships its own libcrypto.so.10 that doesn’t have the GNU symbol version sections.The dynamic loader warns you, but execution usually continues.

You could check if the mongod is seeing that lib from /opt/splunk/lib overridden path:

Simply that error you can ignore as mentioned above.

If loading issue is there: you will see the lib missing

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH ldd $SPLUNK_HOME/bin/mongod

moreover

$ SPLUNK_HOME=/opt/splunk
$ LD_LIBRARY_PATH=$SPLUNK_HOME/lib:$LD_LIBRARY_PATH $SPLUNK_HOME/bin/mongod --version

splunk cmd mongod -version this works for you means no issues with KVstore 7.0

Do you see the process mongod not started by SH?

pgrep -f mongod , can see if its running

also check the kvstore-status

/opt/splunk/bin/splunk show kvstore-status

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

sushilkar — Thu, 14 Aug 2025 05:30:08 GMT

The logs that you see are not harmful, treat it like no version info added library .so file in /opt/lib/splunk its more of a warning. Check if you are able to run mongod --version. (https://community.splunk.com/t5/forums/replypage/board-id/core-splunk-enterprise/message-id/22866)

Check mongod --version you are able to run, see this post.

https://community.splunk.com/t5/Splunk-Enterprise/Issue-with-kvstore-in-version-9-4-3/m-p/751436#M22845

Re: KVstore error output after splunk enterprise upgrade from 9.1.1 to 9.4.1

GSNRMUVW — Thu, 14 Aug 2025 07:31:54 GMT

root@debian:~# /opt/splunk/bin/mongod --version
/opt/splunk/bin/mongod: error while loading shared libraries: libsasl2.so.3: cannot open shared object file: No such file or directory