All Apps and Add-ons

Splunk DB Connect: How to resolve "Can not communicate with task server, check your settings" error message?

hoodaajay1
Engager

I have installed Splunk 6.5.2 enterprise and installed the Splunk DB Connect 3.0 app from the tgz file. When I logon to Splunk and go to Splunk DB Connect, I get an error message at the top that says, 'Can not communicate with task server, check your settings'. I have tried changing the Task server port but that did not help.

Any suggestions/solutions would help.

Thanks

1 Solution

morethanyell
Builder

Make sure KV Store is enabled (splunk show kvstore-status). I spent 4 hours trying to figure out what's wrong with my instance. Nothing in Splunk docs says that Splunk DB Connect requires KV Store to be enabled.

Pcktech
Explorer

Thank you for that heads up. This morning I discovered our ES was lacking the right cert in sslConfig so the splunk-generated cert expired and resulted in https://splunk.my.site.com/customer/s/article/Expired-SSL-certificate-KV-Store-will-not-start-exit-c...

The same fix (adding a serverCert to system/local/server.conf for a PEM containing the Server Cert + Encrypted Private Key) also fixed the KV Store on the HF where DB Connect was installed.

Looks like I'll be verifying sslConfig everywhere in case of other stealthy dependencies on KV Store.

0 Karma

edoardo_vicendo
Contributor

After a system patch JRE path was modified and DB Connect not working anymore with this error in _internal index:

12-03-2021 13:40:51.626 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=dbxquery:  /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/dbxquery.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory

12-03-2021 13:40:51.687 +0100 ERROR ModularInputs - Unable to initialize modular input "dbxquery" defined in the app "splunk_app_db_connect": Introspecting scheme=dbxquery: script running failed (exited with code 126)..

12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - <stderr> Introspecting scheme=server:  /opt/splunk/etc/apps/splunk_app_db_connect/linux_x86_64/bin/server.sh: line 33: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java: No such file or directory

12-03-2021 13:40:52.092 +0100 ERROR ModularInputs - Unable to initialize modular input "server" defined in the app "splunk_app_db_connect": Introspecting scheme=server: script running failed (exited with code 126)..

 

To fix I solved in this way

Before:

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf
[java]
javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre/bin/java

 

After:

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/local > cat dbx_settings.conf
[java]
javaHome = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre

splunk@xxxxxxxx:~/etc/apps/splunk_app_db_connect/linux_x86_64/bin > cat customized.java.path
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.275.b01-0.el6_10.x86_64/jre/bin/java

 

+ Restart splunk

splunkoptimus
Path Finder

I had a similar problem when I upgrade from 3.11.1 to 3.12.2

I had the following error 

12-3-2020 15:44:16.843 +0200 ERROR ExecProcessor [9972 ExecProcessor] - message from "/global/apps/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.sh" action=dbxquery_server_start_failed error=java.lang.RuntimeException: The certificate chain length (11) exceeds the maximum allowed length (10) stack=com.splunk.DBXService.sendImpl(DBXService.java:114)\\com.splunk.DBXService.send(DBXService.java:52)\\com.splunk.HttpService.get(HttpService.java:172)\\com.splunk.Entity.refresh(Entity.java:381)\\com.splunk.Entity.refresh(Entity.java:24)\\com.splunk.Resource.validate(Resource.java:186)\\com.splunk.Entity.validate(Entity.java:462)\\com.splunk.Entity.getContent(Entity.java:157)\\com.splunk.Entity.getString(Entity.java:279)\\com.splunk.ServiceInfo.getVersion(ServiceInfo.java:155)\\com.splunk.DBXService.fetchInfo(DBXService.java:42)\\com.splunk.DBXService.connect(DBXService.java:35)\\com.splunk.dbx.splunkclient.SplunkServiceBuilder.build(SplunkServiceBuilder.java:70)\\com.splunk.dbx.command.DbxQueryServerStart.startDbxQueryServer(DbxQueryServerStart.java:80)\\com.splunk.dbx.command.DbxQueryServerStart.streamEvents(DbxQueryServerStart.java:50)\\com.splunk.modularinput.Script.run(Script.java:66)\\com.splunk.modularinput.Script.run(Script.java:44)\\com.splunk.dbx.command.DbxQueryServerStart.main(DbxQueryServerStart.java:95)\\
 
 
I fixed this problem increasing the certificate chain limit in 
/opt/splunk/etc/apps/splunk_app_db_connect/jars/
server.vmopts
 
parameter: -Djdk.tls.maxCertificateChainLength=13
0 Karma

sujoybose77
Explorer

This was the exact root cause of the issue in my case while upgrading DB connect to version 3.13.

Strangely though, it updated java path correctly in following path:

$SPLUNK_HOME/etc/apps/splunk_app_db_connect/linux_x86/bin

 

I do not know why Splunk DB connect app update process does not take care of updating JAVA path in both linux_x86 and linux_x86_64 location

splunkreal
Motivator

Just installed DBConnect 3.6, used again 1025 for Task Server Port and error appeared. Restarting Splunkd solved the problem (again)

* If this helps, please upvote or accept solution 🙂 *
0 Karma

dhirendra761
Contributor

Hi All,

I have gone though all the answers and done all required changes in config but nothing will happend.
Finally, I use below steps to reslove this issue.

  1. Backup your files, if exist, ex: db_input.conf and drivers
  2. Stop splunk. Remove db_connect app from app folder.
  3. Restart Splunk.
  4. Upgrade your splunk with just new version.
  5. install > db connect app

check your db it will be okay now.

Happy Splunking !..

0 Karma

bingyun
Engager

made sure you had JAVA_HOME set properly to the jre 8 path as well as adding %JAVA_HOME%/bin to the Path

0 Karma

dhihoriya_splun
Splunk Employee
Splunk Employee

There is one file in DBconnect app which is used to configure app:
/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml

There is one stanza in inputs.conf in app's default folder which is used this above file to configure app (including port 9998 for JAVA) so if the inputs are disabled in the app then the app will not able to get the file "/opt/splunk/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml" and it will show the error "Can not communicate with task server, check your settings".

Below is the stanza in inputs.conf:

[server://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
interval = 5
keystore_password = *****
start_by_shell = false

It may be issue with the inputs of DBconnect app.
I have tested above steps and it works fine that by enabling the inputs issue has been resolved.

vnguyen46
Contributor

Hello,
I took me a great amount of time to get to what you shared here. My inputs.conf reads like below. The question is how you enable/disable this .conf file? Does the keystore_password matter (* or changeme)?

Thank you,

[splunk@splunkserver default]$ cat inputs.conf
[server://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbx_task_server.yml
interval = 5
keystore_password = changeme
start_by_shell = false

[dbxquery://default]
config_file = ${SPLUNK_HOME}/etc/apps/splunk_app_db_connect/config/dbxquery_server.yml
interval = 5
keystore_password = changeme
start_by_shell = false

0 Karma

kmower
Communicator

I have uninstalled DB connect completely, and then made sure I had JAVA_HOME set properly to the jre 8 path as well as adding %JAVA_HOME%/bin to the Path. Then I installed DB Connect and it worked. I read this somewhere (sorry probably on answers but I forget the link). But it seems to be essential that JAVA_HOME set along with the reference in the path before installing DB Connect.

kumaranv
Path Finder

Important hint. Set JAVA_HOME first then install DB Connect. It worked.Thanks

0 Karma

jacobpevans
Motivator

Since this is a top hit on Google, I will just add that Java was updated in the background unbeknownst to me. Our auto-updater puts the version in the directory name so I was able to take the above advice, update the Java path for DB Connect and restart to resolve the problem.

Cheers,
Jacob

If you feel this response answered your question, please do not forget to mark it as such. If it did not, but you do have the answer, feel free to answer your own post and accept that as the answer.
0 Karma

splunkreal
Motivator

Using port 1025 solved, thanks guys 🙂

* If this helps, please upvote or accept solution 🙂 *
0 Karma

vnguyen46
Contributor

This is the key to resolve the nasty issue - port 1025.

THANK YOU.

0 Karma

yliang_splunk
Splunk Employee
Splunk Employee

I got the same error for dbx 3.1.4 for splunk 7.1.4 and splunk 6.6.11 in Windows. I installed the brand new splunk and copy paste my old splunk_app_db_connect folder to the splunk/etc/apps/ to 'install' the app. I restart splunk and got the same error message.

My solution is to refresh the Settings under Configuration tab in dbx app. You may need to change the JRE Installation Path by one letter and change it back. Then you click the save on the right. The task server will restart. After a few seconds, it will say task server restart successfully. Then you are ready to go! I didn't change any of the setting but just trigger the refresh task server.

0 Karma

dkretzmer
Explorer

I got this error as well because the java jre path was removed or changed. The Java on my standalone dev Splunk server got updated by our SysAdmins and I was getting the "Can not communicate with task server, check your settings" error. It had been working just fine.
I went to settings and reviewed them. I verified the java path with what was on the server and sure enough the old java path was gone. I updated the java path to the new java path and saved. It attempted the restart of the task server but failed. I restarted splunk and everything came up working like normal.

bzam
Explorer

You may also see this error if you've specified the full path for the JRE Installation Path setting:

Example:
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre

If Java is patched, this will break the task server when Splunk is restarted. If you are using package manager, distributions may maintain symbolic links can be referenced to avoid this breaking often due to patching.

0 Karma

kmower
Communicator

I have this same error on a Windows install. However, I don't have this file at all

C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\default\dbx_task_server.yml

As far as the 'dbx' prefix is concerned I only have dbx_logging and dbx_settings in my C:\Splunk\etc\apps\splunk_app_db_connect\default folder ... this thread is 6.5.3 though, and I am running Splunk 7.1.2 - same error though folks, but no file as referenced here.

Any idea how to fix this? Thanks.

0 Karma

niketn
Legend

Look for the following path:

$SPLUNK_HOME\splunk_app_db_connect\config\dbx_task_server.yml
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...