All Apps and Add-ons

Splunk DB Connect : Connection Refused

edwinmae
Path Finder

Hi,

What does this mean and better how to resolve this issue?

The error below shows up when clicking the RPC service that is UP (green) and Database connection status is also OK

12/30/2015 14:06:16 [ERROR] [init.py] [Errno 111] Connection refused
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 185, in dispatch
methodOutput = method()
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET
self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel
return service.build_service(rpcChannel, max_retries=max_retries, verified=verified)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service
raise error
error: [Errno 111] Connection refused

Tags (1)
0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

If you increase your logging level to debug verbosity you might be getting this error as a log_level=DEBUG in _internal index:

"The RPC server is not ready to serve yet as [Errno 111] Connection refused."

I just looked at the code and that's about the only explanation I have for you. For some reason the RPC connection test is failing so no idea why it shows as "Green". I think you should troubleshoot your RPC server that the db connect app is creating. on the splunk server.

Check out this as it shows how to increase RPC logging verbosity and troubleshoot RPC:

http://docs.splunk.com/Documentation/DBX/2.1.0/DeployDBX/Troubleshooting

alt text

See my comments below about this inputs.conf in db2 and the corresponding rpc port:

View solution in original post

0 Karma

jkat54
SplunkTrust
SplunkTrust

If you increase your logging level to debug verbosity you might be getting this error as a log_level=DEBUG in _internal index:

"The RPC server is not ready to serve yet as [Errno 111] Connection refused."

I just looked at the code and that's about the only explanation I have for you. For some reason the RPC connection test is failing so no idea why it shows as "Green". I think you should troubleshoot your RPC server that the db connect app is creating. on the splunk server.

Check out this as it shows how to increase RPC logging verbosity and troubleshoot RPC:

http://docs.splunk.com/Documentation/DBX/2.1.0/DeployDBX/Troubleshooting

alt text

See my comments below about this inputs.conf in db2 and the corresponding rpc port:

0 Karma

edwinmae
Path Finder

It did the job --
Had to change the Bind IP address to *

Thanks for the support --

0 Karma

jkat54
SplunkTrust
SplunkTrust

@edwinmae if you liked my answer, please mark it as the answer to your question.

0 Karma

chamambom
New Member

@adwinmae ,can you confirm where did you change the Bind IP

0 Karma

edwinmae
Path Finder

/etc/apps/splunk_app_db_connect/local > inputs.conf

edwinmae
Path Finder

alt text


If I set a port lower than 1024, I get a permission denied :

2016-01-05 09:30:40 ERROR RPCServer:230 - Starting RPC server thread failed : java.net.SocketException: Permission denied

The only ports that have been opened are for traffic between Splunk server and 'connected' servers:
1433 (MSSQL) -- Splunk server to Database servers
9997 (Splunk Forwarder) -- Servers to Splunk server


If I set the RPC port to 9997, I no longer get the connection refused:

External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 123 : ''" "

01/05/2016 09:35:35 [ERROR] [__init__.py] '' Traceback (most recent call last): File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 185, in dispatch methodOutput = method() File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host) File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel return service.build_service(rpcChannel, max_retries=max_retries, verified=verified) File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service raise error

With default port 9998:

External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 121 : Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused" "

01/05/2016 09:37:26 [ERROR] [search_command.py] Traceback (most recent call last):
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/splunk_sdk-1.3.1.a-py2.7.egg/splunklib/searchcommands/search_command.py", line 316, in process
    self._execute(operation, reader, writer)
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/splunk_sdk-1.3.1.a-py2.7.egg/splunklib/searchcommands/generating_command.py", line 79, in _execute
    for record in operation():
  File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py", line 121, in generate
    raise Exception("Unable to connect to rpcserver, please check rpc.log: " + error)
Exception: Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused

I assume port 9998 should work but there is probably something else here what causes these problems ...

[root@xxxxxxx xxxxx]# netstat -plnt | grep 9998
tcp        0      0 ::ffff:127.0.0.1:9998       :::*                        LISTEN      13130/java
0 Karma

jkat54
SplunkTrust
SplunkTrust

Yes it is obvious port 9998 is blocked and 9997 is not.

Open TCP port 9998 through the firewall if that's the port you want to use. Your SQL server must be able to contact the RPC server on this port and visa versa.

As to why you cant use a port lower than 1024 this is because only root can open those ports.

edwinmae
Path Finder

Still not working: RPC Service is Up (= green) and DB connections are stated as Valid Connections


  • copied the dbx_logging.conf and log4j.properties to /opt/splunk/etc/apps/splunk_app_db_connect/local
  • level set to DEBUG
  • Restarted Splunk

dbx
01/04/2016 12:32:16 [ERROR] [init.py] [Errno 111] Connection refused
Traceback (most recent call last):
File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/init.py", line 185, in dispatch
methodOutput = method()
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 28, in handle_GET
self.dbx2 = rpc_service_channel(self.userName, self.sessionKey, host=self.host)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/rpcutils.py", line 33, in rpc_service_channel
return service.build_service(rpcChannel, max_retries=max_retries, verified=verified)
File "/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/service.py", line 694, in build_service
raise error
error: [Errno 111] Connection refused

rpc
2016-01-04 12:48:38 DEBUG Connector:153 - DataSource: [jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor:jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor:xxxx:xxxxxxxxx] com.microsoft.sqlserver.jdbc.SQLServerDriver.
2016-01-04 12:48:38 DEBUG Connector:211 - SSL: false
2016-01-04 12:48:38 DEBUG Connector:212 - The database connection information: jdbcDriverClass = [com.microsoft.sqlserver.jdbc.SQLServerDriver], jdbcUrlFormat = [jdbc:sqlserver://xxx.xxx.xxx.xxx:1433;databaseName=xxxx;selectMethod=cursor], username = [xxxxx].
2016-01-04 12:48:38 DEBUG Connector:255 - Database connection was established correctly: jdbcUrlFormat = [jdbc:sqlserver://xxx.xxx.xxx.x:1433;databaseName=xxxxx;selectMethod=cursor], username = [xxxxx].
2016-01-04 12:48:38 INFO Connector:39 - duration=0 splunk-user=nobody connection=null rows=1 params=null sql=SELECT 1
2016-01-04 12:48:38 DEBUG ResultSetConverter:142 - Return Data: NULL..int


When I change the RPC port (number), the error is still the same

I probably have missed something but I don't know what (at the moment) ---

0 Karma

edwinmae
Path Finder

I did also set the ConnectionTimeOut to 300

When trying to run a simple/advanced query:
External search command 'dbxquery' returned error code 1. Script output = " ERROR "Exception at ""/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbxquery.py"", line 121 : Unable to connect to rpcserver, please check rpc.log: [Errno 111] Connection refused" "

0 Karma

jkat54
SplunkTrust
SplunkTrust

What port do you have setup in inputs.conf of the db2 connect app? Is this port open on your server? Is it below port 1024 (if so splunkd has to run as root to open ports lower than 1024)? Is it open on the firewall(s) both software and hardware (iptables, apparmor, windows firewall, etc.)?

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Try going to $SPLUNK_HOME/etc/system/local/web.conf and change your timeout settings as shown below.

 [settings]
 splunkdConnectionTimeout = 300
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...