Splunk DB Connect 2 RPC Service not responding

[christophe_clem]

Hello,

I have an issue with Splunk DB Connect 2 on Splunk 6.3.0, running on Red Hat Enterprise Linux 6.6 x64 : the RPC service seems to be running but is very (very very) slow and almost each request ends up with a timeout.

In the Settings page, I got this message after some seconds:

Splunkd daemon is not responding: ("Error connecting to /servicesNS/nobody/splunk_app_db_connect/db_connect/rpcserver/default?output_mode=json: ('The read operation timed out',)",) 

In the logs, there is the same message, but no other error.

I tried Oracle JRE and openJDK, but the result is the same.
I tried Oracle JRE 8u60 and 8u31. I would like to try JRE i586 but the installation of the RPM fails.

The server has 8 CPU cores and 4 GB of memory.

Kind regards,
Christophe

[acr]

Same issue, did you find a solution ?

[sgundeti]

RPC server runs on port 9998 by default, make sure its not used by any other service. If it is being used by some other service, then edit inputs.conf file in dbx app to change default port to unused port and restart splunk.

[christophe_clem]

Nope, nothing else seems to be listening (or try to) on port TCP/9998.

[bevant]

Have you looked in rpc.log and - more likely - dbx2.log as well?

I assume this happens after you've configured it, so what does the dbx app config look like (inputs.conf)

[christophe_clem]

This is right after I configure it, even the setup page is not working after configuration !

The file inputs.conf contains this:

[rpcstart://default]
javahome = /usr/java/latest/
options = -Xmx1024m -XX:+UseConcMarkSweepGC
useSSL = 0
proc_pid = 3839
Exception = 0

dbx2.log:

10/21/2015 17:03:51 [ERROR] [__init__.py] Socket error communicating with splunkd (error=('The read operation timed out',)), path = /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver
10/21/2015 17:03:51 [ERROR] [__init__.py] Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver: ('The read operation timed out',)",)
Traceback (most recent call last):
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 185, in dispatch
    methodOutput = method()
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 31, in handle_GET
    self.dbx2 = service.build_service(self.get_rpc_channel())
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 19, in get_rpc_channel
    rpc = utils.get_local_rpc(self.userName, self.sessionKey)
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/utils.py", line 170, in get_local_rpc
    ent = en.getEntities(["db_connect", "rpcserver"], namespace=namespace, owner=user, sessionKey=sessionKey)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities
    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed
    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 479, in simpleRequest
    raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver: ('The read operation timed out',)",)
10/21/2015 17:03:51 [ERROR] [__init__.py] Socket error communicating with splunkd (error=('The read operation timed out',)), path = /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver
10/21/2015 17:03:51 [ERROR] [__init__.py] Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver: ('The read operation timed out',)",)
Traceback (most recent call last):
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 185, in dispatch
    methodOutput = method()
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 31, in handle_GET
    self.dbx2 = service.build_service(self.get_rpc_channel())
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbinfo.py", line 19, in get_rpc_channel
    rpc = utils.get_local_rpc(self.userName, self.sessionKey)
  File "/local/opt/splunk/etc/apps/splunk_app_db_connect/bin/dbx2/utils.py", line 170, in get_local_rpc
    ent = en.getEntities(["db_connect", "rpcserver"], namespace=namespace, owner=user, sessionKey=sessionKey)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities
    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed
    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)
  File "/local/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 479, in simpleRequest
    raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))
SplunkdConnectionException: Splunkd daemon is not responding: ("Error connecting to /servicesNS/admin/splunk_app_db_connect/db_connect/rpcserver: ('The read operation timed out',)",)
10/21/2015 17:03:52 [ERROR] [__init__.py] Socket error communicating with splunkd (error=('The read operation timed out',)), path = /servicesNS/nobody/splunk_app_db_connect/db_connect/rpcserver

rpc.log:

2015-10-21 17:03:37 DEBUG ServiceResponder:396 - Request name: getServerStatus.
2015-10-21 17:03:37 DEBUG ServiceResponder:405 - Request parameter(0): class [Ljava.lang.String; = [[HOSTNAME, HOME, SPLUNK_WEB_NAME, SPLUNK_HOME, SPLUNK_SERVER_NAME, TERM_SESSION_ID]].
2015-10-21 17:03:47 DEBUG ServiceResponder:396 - Request name: getServerStatus.
2015-10-21 17:03:47 DEBUG ServiceResponder:405 - Request parameter(0): class [Ljava.lang.String; = [[SPLUNK_HOME]].
2015-10-21 17:03:57 DEBUG ServiceResponder:396 - Request name: getServerStatus.
2015-10-21 17:03:57 DEBUG ServiceResponder:405 - Request parameter(0): class [Ljava.lang.String; = [[HOSTNAME, HOME, SPLUNK_WEB_NAME, SPLUNK_HOME, SPLUNK_SERVER_NAME, TERM_SESSION_ID]].
2015-10-21 17:04:03 DEBUG ServiceResponder:396 - Request name: getServerStatus.
2015-10-21 17:04:03 DEBUG ServiceResponder:405 - Request parameter(0): class [Ljava.lang.String; = [[SPLUNK_HOME]].

[hecorbett]

I'm having similar issues. Did you ever get this resolved?