Why did Task Server fail to start - Splunk DB Connect?

[Splunker_In_Fr]

Dear Community members,

Splunk DB connect on my Splunk Indexer v 9.0.1 is unable to start the Task Server. Based on suggestion related to the error, i looked into the suggestions from many community members and tried most of the them but to no avail. I backed up the configurations and even removed the app and installed it again. Reinstalled JDK and updated the environment variable on the server with correct path for JAVA_HOME and Path variable.

However, when trying to do the set up for Splunk DB Connect, i see the error "Cannot communicate with task server, please check your settings".

When checking on the Configuration>Settings and try to save the settings with JRE Installation, Splunk tries to start the  Task Server but after a minute it times out with error "Failed to start Task Server".

There is not much information on the DB Connect log but on the splunkd logs, i see the following error but unable to understand what could be the reason. We are not using mongodb as a DB and only SQL Server DB is in use to which we wish to connect to. Below the snippet of error from splunkd as it seems to be the root cause of Task Server not starting up.

\windows_x86_64\bin\dbxquery.exe" action=start_dbxquery_server, configFile=D:\Apps\Splunk/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
\windows_x86_64\bin\server.exe" action=start_task_server, configFile=D:\Apps\Splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
\windows_x86_64\bin\dbxquery.exe" 17:35:11.475 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
\windows_x86_64\bin\dbxquery.exe" 17:35:11.553 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store not found, creating
\windows_x86_64\bin\dbxquery.exe" action=dbxquery_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:140)\\com.splunk.DBXService.send(DBXService.java:52)\\com.splunk.HttpService.post(HttpService.java:387)\\com.splunk.EntityCollection.create(EntityCollection.java:95)\\com.splunk.EntityCollection.create(EntityCollection.java:83)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initialize(SecurityFileGenerationUtil.java:245)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:49)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:82)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
\windows_x86_64\bin\dbxquery.exe" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
\windows_x86_64\bin\dbxquery.exe" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:51)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
\windows_x86_64\bin\server.exe" 17:35:11.694 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - initializing secret kv store collection
\windows_x86_64\bin\server.exe" 17:35:11.741 [main] INFO com.splunk.dbx.utils.SecurityFileGenerationUtil - secret KV Store not found, creating
\windows_x86_64\bin\server.exe" action=task_server_start_failed error=com.splunk.HttpException: HTTP 503 -- KV Store initialization failed. Please contact your system administrator. stack=com.splunk.HttpException.create(HttpException.java:84)\\com.splunk.DBXService.sendImpl(DBXService.java:140)\\com.splunk.DBXService.send(DBXService.java:52)\\com.splunk.HttpService.post(HttpService.java:387)\\com.splunk.EntityCollection.create(EntityCollection.java:95)\\com.splunk.EntityCollection.create(EntityCollection.java:83)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initialize(SecurityFileGenerationUtil.java:245)\\com.splunk.dbx.utils.SecurityFileGenerationUtil.initEncryption(SecurityFileGenerationUtil.java:49)\\com.splunk.dbx.server.bootstrap.TaskServerStart.startTaskServer(TaskServerStart.java:108)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:69)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
\windows_x86_64\bin\server.exe" com.splunk.modularinput.MalformedDataException: Events must have at least the data field set to be written to XML.
\windows_x86_64\bin\server.exe" com.splunk.modularinput.Event.writeTo(Event.java:65)\\com.splunk.modularinput.EventWriter.writeEvent(EventWriter.java:137)\\com.splunk.dbx.server.bootstrap.TaskServerStart.streamEvents(TaskServerStart.java:74)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.server.bootstrap.TaskServerStart.main(TaskServerStart.java:145)\\
\windows_x86_64\bin\server.exe" action=start_task_server, configFile=D:\Apps\Splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml

Thanks for your help!

[splunkoptimus]

Check if the KV store is running (splunk show kvstore status), if not start it. 

[deepthi5]

Hi ,Did you get a solution to this 

[isoutamo]

Hi

If you are running DBX version 3.13.0 there is a new feature to run DBX as HA mode on SHC. This probably needs kvstore as all inputs/outputs are running on captain. See: https://docs.splunk.com/Documentation/DBX/3.13.0/ReleaseNotes/Releasenotes

Basically now you could run HA version of DBX as creating a separate "SHC" from your DBX HFs 😉

Have you a running kvstore on your node?

r. Ismo