topic Re: Splunk DB Connect : Connection Refused in All Apps and Add-ons

Splunk DB Connect : Connection Refused

edwinmae — Tue, 29 Sep 2020 08:13:56 GMT

Hi,

What does this mean and better how to resolve this issue?

The error below shows up when clicking the RPC service that is UP (green) and Database connection status is also OK

12/30/2015 14:06:16 [ERROR] [init.py] [Errno 111] Connection refused
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 185, in dispatch
methodOutput = method()
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET
self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel
return service.build_service(rpcChannel, max_retries=max_retries, verified=verified)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service
raise error
error: [Errno 111] Connection refused

Re: Splunk DB Connect : Connection Refused

jkat54 — Fri, 01 Jan 2016 14:26:57 GMT

If you increase your logging level to debug verbosity you might be getting this error as a log_level=DEBUG in _internal index:

"The RPC server is not ready to serve yet as [Errno 111] Connection refused."

I just looked at the code and that's about the only explanation I have for you. For some reason the RPC connection test is failing so no idea why it shows as "Green". I think you should troubleshoot your RPC server that the db connect app is creating. on the splunk server.

Check out this as it shows how to increase RPC logging verbosity and troubleshoot RPC:

http://docs.splunk.com/Documentation/DBX/2.1.0/DeployDBX/Troubleshooting

See my comments below about this inputs.conf in db2 and the corresponding rpc port:

Re: Splunk DB Connect : Connection Refused

jcoates_splunk — Sun, 03 Jan 2016 01:35:56 GMT

Try going to $SPLUNK_HOME/etc/system/local/web.conf and change your timeout settings as shown below.

 [settings]
 splunkdConnectionTimeout = 300

Re: Splunk DB Connect : Connection Refused

edwinmae — Tue, 29 Sep 2020 08:18:09 GMT

Still not working: RPC Service is Up (= green) and DB connections are stated as Valid Connections

copied the dbx_logging.conf and log4j.properties to /opt/splunk/etc/apps/splunk_app_db_connect/local
level set to DEBUG
Restarted Splunk

dbx
01/04/2016 12:32:16 [ERROR] [init.py] [Errno 111] Connection refused
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 185, in dispatch
methodOutput = method()
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET
self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel
return service.build_service(rpcChannel, max_retries=max_retries, verified=verified)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service
raise error
error: [Errno 111] Connection refused

rpc
2016-01-04 12:48:38 DEBUG Connector:153 - DataSource: [jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor:jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor:xxxx:xxxxxxxxx] com.microsoft.sqlserver.jdbc.SQLServerDriver.
2016-01-04 12:48:38 DEBUG Connector:211 - SSL: false
2016-01-04 12:48:38 DEBUG Connector:212 - The database connection information: jdbcDriverClass = [com.microsoft.sqlserver.jdbc.SQLServerDriver], jdbcUrlFormat = [jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor], username = [xxxxx].
2016-01-04 12:48:38 DEBUG Connector:255 - Database connection was established correctly: jdbcUrlFormat = [jdbc:sqlserver://xxx.xxx.xxx.x:1433;databaseName=xxxxx;selectMethod=cursor], username = [xxxxx].
2016-01-04 12:48:38 INFO Connector:39 - duration=0 splunk-user=nobody connection=null rows=1 params=null sql=SELECT 1
2016-01-04 12:48:38 DEBUG ResultSetConverter:142 - Return Data: NULL..int

When I change the RPC port (number), the error is still the same

I probably have missed something but I don't know what (at the moment) ---

Re: Splunk DB Connect : Connection Refused

edwinmae — Tue, 29 Sep 2020 08:18:13 GMT

I did also set the ConnectionTimeOut to 300

When trying to run a simple/advanced query:
External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 121 : Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused" "

Re: Splunk DB Connect : Connection Refused

jkat54 — Mon, 04 Jan 2016 12:41:52 GMT

What port do you have setup in inputs.conf of the db2 connect app? Is this port open on your server? Is it below port 1024 (if so splunkd has to run as root to open ports lower than 1024)? Is it open on the firewall(s) both software and hardware (iptables, apparmor, windows firewall, etc.)?

Re: Splunk DB Connect : Connection Refused

edwinmae — Tue, 05 Jan 2016 08:10:56 GMT

If I set a port lower than 1024, I get a permission denied :

2016-01-05 09:30:40 ERROR RPCServer:230 - Starting RPC server thread failed : java.net.SocketException: Permission denied

The only ports that have been opened are for traffic between Splunk server and 'connected' servers:
1433 (MSSQL) -- Splunk server to Database servers
9997 (Splunk Forwarder) -- Servers to Splunk server

If I set the RPC port to 9997, I no longer get the connection refused:

External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 123 : ''" "

01/05/2016 09:35:35 [ERROR] [__init__.py] '' Traceback (most recent call last): File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 185, in dispatch methodOutput = method() File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host) File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel return service.build_service(rpcChannel, max_retries=max_retries, verified=verified) File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service raise error

With default port 9998:

External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 121 : Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused" "

01/05/2016 09:37:26 [ERROR] [search_command.py] Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/splunk_sdk-1.3.1.a-py2.7.egg/splunklib/searchcommands/search_command.py", line 316, in process
    self._execute(operation, reader, writer)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/splunk_sdk-1.3.1.a-py2.7.egg/splunklib/searchcommands/generating_command.py", line 79, in _execute
    for record in operation():
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py", line 121, in generate
    raise Exception("Unable to connect to rpcserver, please check rpc.log: " + error)
Exception: Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused

I assume port 9998 should work but there is probably something else here what causes these problems ...

[root@xxxxxxx xxxxx]# netstat -plnt | grep 9998
tcp        0      0 ::ffff:127.0.0.1:9998       :::*                        LISTEN      13130/java

Re: Splunk DB Connect : Connection Refused

jkat54 — Wed, 06 Jan 2016 12:37:28 GMT

Yes it is obvious port 9998 is blocked and 9997 is not.

Open TCP port 9998 through the firewall if that's the port you want to use. Your SQL server must be able to contact the RPC server on this port and visa versa.

As to why you cant use a port lower than 1024 this is because only root can open those ports.

Re: Splunk DB Connect : Connection Refused

edwinmae — Wed, 13 Jan 2016 08:56:39 GMT

It did the job --
Had to change the Bind IP address to *

Thanks for the support --

Re: Splunk DB Connect : Connection Refused

jkat54 — Sun, 17 Jan 2016 19:09:38 GMT

@edwinmae if you liked my answer, please mark it as the answer to your question.

Re: Splunk DB Connect : Connection Refused

chamambom — Fri, 11 Nov 2016 12:48:27 GMT

@adwinmae ,can you confirm where did you change the Bind IP

Re: Splunk DB Connect : Connection Refused

edwinmae — Tue, 29 Sep 2020 11:45:07 GMT

/etc/apps/splunk_app_db_connect/local > inputs.conf