All Apps and Add-ons

Splunk DB Connect: How to resolve RPC Server claiming to be on a conflicting port resulting in a restart fail loop?

New Member

I'm running Splunk DB Connect 2.4.0 with drivers for drivers for Postgres 9.4 and Redshift. The RPC Server seems to start up but then shuts down. While it is up I can make some queries, after which I then get errors like this in Splunk Web.

"External search command 'dbxquery' returned error code 1. Script output = "RuntimeError: Failed to run query:  caused by: error(111, 'Connection refused')."

Here is the rpc.log (about 5 minutes later) this log will repeat.

2016-12-07 23:23:10 INFO  RPCServer:71 - action=rpc_server_is_starting_up args=[127.0.0.1:9998]
2016-12-07 23:23:11 DEBUG RPCServer:147 - action=rpc_server_protocol_parsed
2016-12-07 23:23:11 DEBUG RPCServer:208 - action=initialize_rpc_server_with_non_ssl_connector
2016-12-07 23:23:11 DEBUG RPCServer:289 - action=service_servlets_registered endpoint=[/dbx2/*, /ws/*]
2016-12-07 23:23:11 DEBUG RPCServer:297 - action=test_servlet_in_debug_mode_registered endpoint=/test/*
2016-12-07 23:23:11 INFO  RPCServer:222 - action=rpc_server_starts_to_listening bind_ip=127.0.0.1 port=9998 use_ssl=false
2016-12-07 23:23:11 DEBUG RPCServer:302 - action=rpc_server_is_starting
2016-12-07 23:23:11 DEBUG RPCServer:304 - action=rpc_server_is_dumping_beans
2016-12-07 23:23:11 DEBUG RPCServer:306 - action=rpc_server_thread_is_joining
2016-12-07 23:23:13 DEBUG ServiceResponder:309 - action=process_request request=getDriverInfo
2016-12-07 23:23:13 DEBUG DBX2Proxy:89 - Informix driver not found
2016-12-07 23:23:13 DEBUG DBX2ServiceFactory:26 - action=try_to_instansiate_service service=com.splunk.dbx2.DefaultDBX2JDBC
2016-12-07 23:23:13 DEBUG DBX2ServiceFactory:26 - action=try_to_instansiate_service service=com.splunk.dbx2.DefaultDBX2JDBC
2016-12-07 23:23:13 DEBUG DBX2ServiceFactory:32 - action=service_instansiated service=com.splunk.dbx2.DefaultDBX2JDBC
2016-12-07 23:23:13 DEBUG DBX2ServiceFactory:32 - action=service_instansiated service=com.splunk.dbx2.DefaultDBX2JDBC
2016-12-07 23:25:06 DEBUG RPCServer:97 - action=process_rpc_server_jvm_shutdown_hook
2016-12-07 23:25:06 INFO  RPCServer:79 - action=rpc_server_is_called_to_halt
2016-12-07 23:25:06 INFO  RPCServer:83 - action=rpc_server_is_halted
2016-12-07 23:25:06 DEBUG RPCServer:88 - action=rpc_server_is_destroyed
2016-12-07 23:25:07 INFO  RPCServer:71 - action=rpc_server_is_starting_up args=[127.0.0.1:9998]
2016-12-07 23:25:07 DEBUG RPCServer:147 - action=rpc_server_protocol_parsed
2016-12-07 23:25:07 DEBUG RPCServer:208 - action=initialize_rpc_server_with_non_ssl_connector
2016-12-07 23:25:07 DEBUG RPCServer:289 - action=service_servlets_registered endpoint=[/dbx2/*, /ws/*]
2016-12-07 23:25:07 DEBUG RPCServer:297 - action=test_servlet_in_debug_mode_registered endpoint=/test/*
2016-12-07 23:25:07 INFO  RPCServer:222 - action=rpc_server_starts_to_listening bind_ip=127.0.0.1 port=9998 use_ssl=false
2016-12-07 23:25:07 DEBUG RPCServer:302 - action=rpc_server_is_starting
2016-12-07 23:25:07 ERROR RPCServer:228 - action=rpc_server_port_is_in_use port=9998 Please specify a new port for RPC server to run on.
2016-12-07 23:25:07 DEBUG RPCServer:97 - action=process_rpc_server_jvm_shutdown_hook
2016-12-07 23:25:07 INFO  RPCServer:79 - action=rpc_server_is_called_to_halt
2016-12-07 23:25:07 INFO  RPCServer:83 - action=rpc_server_is_halted
2016-12-07 23:25:07 DEBUG RPCServer:88 - action=rpc_server_is_destroyed

and I'm running java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)

I have tried changing ports and am striking out on any Google, Splunk Answers searches... please help

0 Karma

New Member

Turns out that the loop was being caused the Addon Debug Refresh, which defaults to run on a 5 minute cron. Disabling the cron on that add on fixed the loop aspect of this problem.

However, the problem where DBConnect doesn't restart correctly still persists. When we refresh, we have to wait a minute or two so the RPC service is actually down and then refresh again to get it to start up again.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!