Splunkd stop responding after the event presented below.
Splunk Tech. Support filed a bug against Splunk Enterprise version 6.0.1.
02-12-2014 13:59:01.768 +0000 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 527, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/system/bin/DataModelAccelerationHandler.py", line 20, in handleList\n sc_rest.BaseRestHandler.handleList(self, confInfo)\n File "/opt/splunk/etc/system/bin/sc_rest.py", line 74, in handleList\n ent = self.all()\n File "/opt/splunk/etc/system/bin/sc_rest.py", line 221, in all\n offset=self.posOffset)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities\n atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed\n serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 469, in simpleRequest\n raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))\nSplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /servicesNS/-/-/data/models: _ssl.c:506: The handshake operation timed out',)\n
This issue was identified as deadlock bug in openssl.
I had a similar problem when I tried to use SplunkDBConnect (dbx) app, i had similar errors (which were actually a missing mysql driver problem) and the jbridge was hanging all the time.
If you have DB inputs, check that you have the correct driver for DB engine:
Also, in java.conf for dbx app you might want to add the bridge stanza:
addr = 127.0.0.1
port = XXX
threads = 10
This worked for me!
We are still being affected by this issue. We have captured pstacks. Splunk tech support identified a dead lock bug in openssl. However, the issue is not fixed in Splunk 6.1.2. Are you still being affected by this issue?