Knowledge Management
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Ask a Question

HTTP 503 -- KV Store initialization failed. Please contact your system administrator

Anubaatar
Loves-to-Learn Everything
‎06-21-2024 11:59 PM

Hi,

Iam having this error since first of the june. Here is my splunkd.log

 

06-22-2024 14:54:00.405 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:00.406 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:00.406 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:51)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:04.800 +0800 INFO ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" action=start_task_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
06-22-2024 14:54:04.842 +0800 INFO ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=start_dbxquery_server, configFile=/opt/splunk/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
06-22-2024 14:54:04.981 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" 14:54:04.980 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
06-22-2024 14:54:05.015 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" 14:54:05.013 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
06-22-2024 14:54:05.102 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" 14:54:05.101 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store found, store=com.splunk.Entity@d7b1517
06-22-2024 14:54:05.129 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" 14:54:05.129 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store found, store=com.splunk.Entity@d7b1517
06-22-2024 14:54:05.214 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" action=task_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.server.bootstrap.TaskServerStart.startTaskServer(TaskServerStart.java:108)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:69)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
06-22-2024 14:54:05.215 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:05.215 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:74)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
06-22-2024 14:54:05.233 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:132)\\com.splunk.DBXService.send(DBXService.java:44)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.dbx.model.repository.SecretKVStoreRepository.getSecrets(SecretKVStoreRepository.java:41)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.getSecretsFromKvStore(SecurityFileGenerationUtil.java:261)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:51)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
06-22-2024 14:54:05.233 +0800 WARN ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
06-22-2024 14:54:05.233 +0800 ERROR ExecProcessor [201562 ExecProcessor] - message from "/opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:51)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\



And here is the mongod.log

2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslPEMKeyPassword is deprecated. Please use tlsCertificateKeyFilePassword instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
2024-06-19T15:46:17.512+0800 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
2024-06-19T07:46:17.513Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
2024-06-19T07:46:17.522Z W NETWORK [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting
2024-06-19T07:46:17.524Z W ASIO [main] No TransportLayer configured during NetworkInterface startup
2024-06-19T07:46:17.527Z I ACCESS [main] permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open



I tried create new ssl certificate but it doesnt work. And tried change the permission of the
/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
still encountering same error.

What should i do? Please help.

Labels (1)
Labels
0 Karma
Reply

Anubaatar
Loves-to-Learn Everything
‎06-23-2024 06:22 PM

i think its not about
"permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open"

i tried  permission 400 and 600 and user group is both splunk. What should i do? Please help me.

 

0 Karma
Reply

tscroggins
Influencer
‎06-23-2024 09:18 AM

If you still see "permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open," make sure the file is owned by your Splunk user and change the permissions to user (owner) read or read+write:

$ chmod 0600 /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key

If you're using a file system that supports extended ACLs, also make sure none are applied. You can check with getfacl:

$ getfacl -p /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
# file: /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key
# owner: splunk
# group: splunk
user::rw-
group::---
other::---

 

0 Karma
Reply

Anubaatar
Loves-to-Learn Everything
‎06-23-2024 07:34 PM

i think its not about
"permissions on /opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key are too open"

i tried  permission 400 and 600 and user group is both splunk. What should i do? Please help me.

0 Karma
Reply
Get Updates on the Splunk Community!

Meet Duke Cyberwalker | A hero’s journey with Splunk

We like to say, the lightsaber is to Luke as Splunk is to Duke. Curious yet? Then read Eric Fusilero’s latest ...

The Future of Splunk Search is Here - See What’s New!

We’re excited to introduce two powerful new search features, now generally available for Splunk Cloud Platform ...

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...