Splunk DB Connect: How to resolve "Can not communicate with task server, check your settings" error message?

[hoodaajay1]

I have installed Splunk 6.5.2 enterprise and installed the Splunk DB Connect 3.0 app from the tgz file. When I logon to Splunk and go to Splunk DB Connect, I get an error message at the top that says, 'Can not communicate with task server, check your settings'. I have tried changing the Task server port but that did not help.

Any suggestions/solutions would help.

Thanks

[kmower]

Thanks. You mean JAVA_HOME in that file, or something else? Thanks again, I really appreciate it.

[kmower]

The top of my dbx_task_server.yml reads:

server:
rootPath: /api/
applicationConnectors:
- type: http
port: 9998

I can't find any /api/ path in the Splunk directories. Perhaps this is the issue?

[jayanath]

I had the same problem and search using index=_internal sourcetype="dbx"* gave out the real error which was a permission issue.

[roopeshiiitm]

Error : Task Server not started

first we keep port 1025 (or 1030) any port save it this will still not working
restart splunk

we can see splunk db connect working fine , go to configuration you will see its using the port which
we have given like 1025 in our case

now we can change port again to 9998 , save and task server working fine

done ,first we keep port 1025 (or 1030 any port ) save , this will still show task server not started
now

restart splunk

we found its working now , however port is what we have given , like 1025 in our case

now we can change port again to 9998 , and can see task server working fine

Done

[kacel]

its works for me thank u .

[altink]

Hi

I did the change to 1025 - it didn't work. and I turned it again to 9998 - same result.
In both cases, when I start Splunk I have the following output:

[splunk@srvsplunk bin]$ ./splunk start

 Splunk> CSI: Logfiles.

 Checking prerequisites...
     Checking http port [8000]: open
     Checking mgmt port [8089]: open
     Checking appserver port [127.0.0.1:8065]: open
     Checking kvstore port [8191]: open
     Checking configuration...  Done.
     Checking critical directories...    Done
     Checking indexes...
         Validated: _audit _internal _introspection _telemetry _thefishbucket history main omega_ca_nt omega_ds omega_ds_idx_01 omega_nt_ora summary
     Done
     Checking filesystem compatibility...  Done
     Checking conf files for problems...
         Invalid key in stanza [dbx_java_installation_health_check] in /opt/splunk/etc/apps/splunk_app_db_connect/default/checklist.conf, line 17: doc_title  (value:  DB Connect JVM installation).
         Invalid key in stanza [dbx_driver_installation_health_check] in /opt/splunk/etc/apps/splunk_app_db_connect/default/checklist.conf, line 41: doc_title  (value:  DB Connect JDBC Driver installation).
         Invalid key in stanza [dbx_connections_configuration_health_check] in /opt/splunk/etc/apps/splunk_app_db_connect/default/checklist.conf, line 69: doc_title  (value:  DB Connect connection configuration,DB Connect identity configuration).
         Invalid key in stanza [dbx_data_lab_configuration_health_check] in /opt/splunk/etc/apps/splunk_app_db_connect/default/checklist.conf, line 85: doc_title  (value:  DB Connect input configuration,DB Connect output configuration,DB Connect lookup configuration).
         Invalid key in stanza [dbx_java_server_configuration_health_check] in /opt/splunk/etc/apps/splunk_app_db_connect/default/checklist.conf, line 98: doc_title  (value:  DB Connect Java Server configuration).
         Your indexes and inputs configurations are not internally consistent. For more information, run 'splunk btool check --debug'
     Done
     Checking default conf files for edits...
     Validating installed files against hashes from '/opt/splunk/splunk-6.5.2-67571ef4b87d-linux-2.6-x86_64-manifest'
     All installed files intact.
     Done
 All preliminary checks passed.

 Starting splunk server daemon (splunkd)...  
 Done
  [  OK  ]

 Waiting for web server at http://127.0.0.1:8000 to be available........ Done


 If you get stuck, we're here to help.  
 Look for answers here: http://docs.splunk.com

 The Splunk web interface is at http://srvsplunk:8000

the very strange thing on the above - is that the startup problems DO NOT appear with the 9998 port as an original setting after the DB Connect App Install, but they DO appear even when the port is set again to 9998 (after trying 1025).

regards
Altin

[jcoates_splunk · Accepted solution]

http://docs.splunk.com/Documentation/DBX/3.0.1/DeployDBX/Troubleshooting#DB_Connect_logging

what's the error message?

[hoodaajay1]

This was the error message.
Can not communicate with task server, check your settings

From the Settings screen on the webui for Db Connect I was not able to change the task server port as after the start it tries to restart the task server & it was failing.

I changed the port setting in this file from 9998 to 1025 & restarted Splunk from services, it worked for me.
C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\default\dbx_task_server.yml
server:
rootPath: /api/
applicationConnectors:
- type: https
port: 1025

[rogue_carrot]

Yes, If I change the port number and restart splunk it seems to work! Thank-you for the helpful answer.

./splunk restart

[vsilchev]

Search index=_internal sourcetype="dbx*" error gives me events like this:

2017-03-15 13:58:05.813 +0400 3764@TestWin2012 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down
java.lang.RuntimeException: java.net.BindException: Address already in use: bind
    at org.eclipse.jetty.setuid.SetUIDListener.lifeCycleStarting(SetUIDListener.java:213)
    ...

So, it seems like DB Connect cannot start task server using port 9998 because this port number is already occupied.

Changing port setting actually solves the problem. However, it is still unclear what could cause it. For me this problem appeared right after I upgraded DB Connect from 2.4 to 3.0.

[mikkorh]

Yes that's the only reason why DB Connect cannot occupy the port if something else is already listening there. Nothing should be listening in that port unless configured unless Splunk for some reason reserves that port for what ever reason (SSL?) by default. I think it's a 'feature' which has slipped through QA fingers.

[mikkorh]

Hoodaajay1, your answer fixed the same error situation also in my case. Haven't yet tested this DB Connect 3 in linux but at least in windows installation it works and looks like it survives at least one reboot.

Thank you!

[apezuela]

Check unix permissions over app bin folder, scripts need exec right (+x).