Splunk DB Connect: How to resolve "Can not communicate with task server, check your settings" error message?

[hoodaajay1]

I have installed Splunk 6.5.2 enterprise and installed the Splunk DB Connect 3.0 app from the tgz file. When I logon to Splunk and go to Splunk DB Connect, I get an error message at the top that says, 'Can not communicate with task server, check your settings'. I have tried changing the Task server port but that did not help.

Any suggestions/solutions would help.

Thanks

[jokertothequinn]

This is what I found and it worked!

First of all the message, "Can not communicate with task server......" is vague and does not give a clear idea so it can be so many reasons, few of them are
1. Updating the java_home path
2. checking the jre version
3. Checking if the HF has approved license or is connected to the License Manager (its no longer the license master)
4. Change the task server port to 9995 or 1025, instead of 9998

What I Did was this:
in the SPLUNK_HOME/var/log/splunk/splunkd.log  is showed some error for dbx-migration.conf
so i added these lines by creating dbx-migration.conf in /etc/apps/splunk_app_db_connect/local

[encryption]
disabled = 0
upgrade = DONE

Then a restart of splunkd.
Works super smooth

[morethanyell]

Make sure KV Store is enabled (splunk show kvstore-status). I spent 4 hours trying to figure out what's wrong with my instance. Nothing in Splunk docs says that Splunk DB Connect requires KV Store to be enabled.

[rvany]

Short documentation reminder: https://docs.splunk.com/Documentation/DBX/3.18.2/DeployDBX/Prerequisites

"KV store must also be active and working properly as of DB Connect version 3.10.0 and higher"

[Pcktech]

Thank you for that heads up. This morning I discovered our ES was lacking the right cert in sslConfig so the splunk-generated cert expired and resulted in https://splunk.my.site.com/customer/s/article/Expired-SSL-certificate-KV-Store-will-not-start-exit-c...

The same fix (adding a serverCert to system/local/server.conf for a PEM containing the Server Cert + Encrypted Private Key) also fixed the KV Store on the HF where DB Connect was installed.

Looks like I'll be verifying sslConfig everywhere in case of other stealthy dependencies on KV Store.

[edoardo_vicendo]

After a system patch JRE path was modified and DB Connect not working anymore with this error in _internal index:

12-03-2021 13:40:51.626 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=dbxquery:  /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory

12-03-2021 13:40:51.687 +0100 ERROR ModularInputs - Unable to initialize modular input "dbxquery" defined in the app "splunk_app_db_connect": Introspecting scheme=dbxquery: script running failed (exited with code 126)..

12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=server:  /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory

12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - Unable to initialize modular input "server" defined in the app "splunk_app_db_connect": Introspecting scheme=server: script running failed (exited with code 126)..

To fix I solved in this way

Before:

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf
[java]
javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java

After:

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf
[java]
javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre/bin/java

+ Restart splunk

[splunkoptimus]

I had a similar problem when I upgrade from 3.11.1 to 3.12.2

I had the following error 

12-3-2020 15:44:16.843 +0200 ERROR ExecProcessor [9972 ExecProcessor] - message from "/global/apps/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" action=dbxquery_server_start_failed error=java.lang.RuntimeException: The certificate chain length (11) exceeds the maximum allowed length (10) stack=com.splunk.DBXService.sendImpl(DBXService.java:114)\\com.splunk.DBXService.send(DBXService.java:52)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.Entity.refresh(Entity.java:381)\\com.splunk.Entity.refresh(Entity.java:24)\\com.splunk.Resource.validate(Resource.java:186)\\com.splunk.Entity.validate(Entity.java:462)\\com.splunk.Entity.getContent(Entity.java:157)\\com.splunk.Entity.getString(Entity.java:279)\\com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)\\com.splunk.DBXService.fetchInfo(DBXService.java:42)\\com.splunk.DBXService.connect(DBXService.java:35)\\com.splunk.dbx.splunkclient.SplunkServiceBuilder.build(SplunkServiceBuilder.java:70)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:80)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\

I fixed this problem increasing the certificate chain limit in 

/opt/splunk/etc/apps/splunk_app_db_connect/jars/

server.vmopts

parameter: -Djdk.tls.maxCertificateChainLength=13

[sujoybose77]

This was the exact root cause of the issue in my case while upgrading DB connect to version 3.13.

Strangely though, it updated java path correctly in following path:

$SPLUNK_HOME/etc/apps/splunk_app_db_connect/linux_x86/bin

I do not know why Splunk DB connect app update process does not take care of updating JAVA path in both linux_x86 and linux_x86_64 location

[splunkreal]

Just installed DBConnect 3.6, used again 1025 for Task Server Port and error appeared. Restarting Splunkd solved the problem (again)

[dhirendra761]

Hi All,

I have gone though all the answers and done all required changes in config but nothing will happend.
Finally, I use below steps to reslove this issue.

1. Backup your files, if exist, ex: db_input.conf and drivers
2. Stop splunk. Remove db_connect app from app folder.
3. Restart Splunk.
4. Upgrade your splunk with just new version.
5. install > db connect app

check your db it will be okay now.

Happy Splunking !..

[bingyun]

made sure you had JAVA_HOME set properly to the jre 8 path as well as adding %JAVA_HOME%/bin to the Path

[dhihoriya_splun]

There is one file in DBconnect app which is used to configure app:
/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml

There is one stanza in inputs.conf in app's default folder which is used this above file to configure app (including port 9998 for JAVA) so if the inputs are disabled in the app then the app will not able to get the file "/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml" and it will show the error "Can not communicate with task server, check your settings".

Below is the stanza in inputs.conf:

[server://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
interval = 5
keystore_password = *****
start_by_shell = false

It may be issue with the inputs of DBconnect app.
I have tested above steps and it works fine that by enabling the inputs issue has been resolved.

[vnguyen46]

Hello,
I took me a great amount of time to get to what you shared here. My inputs.conf reads like below. The question is how you enable/disable this .conf file? Does the keystore_password matter (* or changeme)?

Thank you,

[splunk@splunkserver default]$ cat inputs.conf
[server://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
interval = 5
keystore_password = changeme
start_by_shell = false

[dbxquery://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
interval = 5
keystore_password = changeme
start_by_shell = false

[kmower]

I have uninstalled DB connect completely, and then made sure I had JAVA_HOME set properly to the jre 8 path as well as adding %JAVA_HOME%/bin to the Path. Then I installed DB Connect and it worked. I read this somewhere (sorry probably on answers but I forget the link). But it seems to be essential that JAVA_HOME set along with the reference in the path before installing DB Connect.

[kumaranv]

Important hint. Set JAVA_HOME first then install DB Connect. It worked.Thanks

[jacobpevans]

Since this is a top hit on Google, I will just add that Java was updated in the background unbeknownst to me. Our auto-updater puts the version in the directory name so I was able to take the above advice, update the Java path for DB Connect and restart to resolve the problem.

[splunkreal]

Using port 1025 solved, thanks guys 🙂

[vnguyen46]

This is the key to resolve the nasty issue - port 1025.

THANK YOU.

[yliang_splunk]

I got the same error for dbx 3.1.4 for splunk 7.1.4 and splunk 6.6.11 in Windows. I installed the brand new splunk and copy paste my old splunk_app_db_connect folder to the splunk/etc/apps/ to 'install' the app. I restart splunk and got the same error message.

My solution is to refresh the Settings under Configuration tab in dbx app. You may need to change the JRE Installation Path by one letter and change it back. Then you click the save on the right. The task server will restart. After a few seconds, it will say task server restart successfully. Then you are ready to go! I didn't change any of the setting but just trigger the refresh task server.

[dkretzmer]

I got this error as well because the java jre path was removed or changed. The Java on my standalone dev Splunk server got updated by our SysAdmins and I was getting the "Can not communicate with task server, check your settings" error. It had been working just fine.
I went to settings and reviewed them. I verified the java path with what was on the server and sure enough the old java path was gone. I updated the java path to the new java path and saved. It attempted the restart of the task server but failed. I restarted splunk and everything came up working like normal.

[bzam]

You may also see this error if you've specified the full path for the JRE Installation Path setting:

Example:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre

If Java is patched, this will break the task server when Splunk is restarted. If you are using package manager, distributions may maintain symbolic links can be referenced to avoid this breaking often due to patching.