HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Anubaatar — Sat, 22 Jun 2024 06:59:36 GMT

Hi,

Iam having this error since first of the june. Here is my splunkd.log

06-22-2024 14:54:00.405 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:00.406 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:00.406 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:51)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:04.800 +0800 INFO ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" action=start_task_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
06-22-2024 14:54:04.842 +0800 INFO ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=start_dbxquery_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
06-22-2024 14:54:04.981 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" 14:54:04.980 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
06-22-2024 14:54:05.015 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" 14:54:05.013 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
06-22-2024 14:54:05.102 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" 14:54:05.101 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store found, store=com.splunk.Entity@d7b1517
06-22-2024 14:54:05.129 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" 14:54:05.129 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store found, store=com.splunk.Entity@d7b1517
06-22-2024 14:54:05.214 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" action=task_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.server.bootstrap.TaskServerStart.startTaskServer(TaskServerStart.java:108)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:69)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
06-22-2024 14:54:05.215 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:05.215 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:74)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
06-22-2024 14:54:05.233 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:05.233 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:05.233 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:51)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\

And here is the mongod.log

2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslPEMKeyPassword is deprecated. Please use tlsCertificateKeyFilePassword instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
2024-06-19T07:46:17.513Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
2024-06-19T07:46:17.522Z W NETWORK [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting
2024-06-19T07:46:17.524Z W ASIO [main] No TransportLayer configured during NetworkInterface startup
2024-06-19T07:46:17.527Z I ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open

I tried create new ssl certificate but it doesnt work. And tried change the permission of the
/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
still encountering same error.

What should i do? Please help.

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

tscroggins — Sun, 23 Jun 2024 16:29:09 GMT

If you still see "permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open," make sure the file is owned by your Splunk user and change the permissions to user (owner) read or read+write:

$ chmod 0600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

If you're using a file system that supports extended ACLs, also make sure none are applied. You can check with getfacl:

$ getfacl -p /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key # file: /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key # owner: splunk # group: splunk user::rw- group::--- other::---

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Anubaatar — Mon, 24 Jun 2024 01:22:22 GMT

i think its not about
"permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open"

i tried permission 400 and 600 and user group is both splunk. What should i do? Please help me.

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Anubaatar — Mon, 24 Jun 2024 02:34:10 GMT

topic Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator in Knowledge Management

HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Re: HTTP 503 -- KV Store initialization failed. Please contact your system administrator