splunkd stop responding -> ERROR AdminManager

[lpolo]

Splunkd stop responding after the event presented below.

Splunk Tech. Support filed a bug against Splunk Enterprise version 6.0.1.

02-12-2014 13:59:01.768 +0000 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 527, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/system/bin/DataModelAccelerationHandler.py", line 20, in handleList\n    sc_rest.BaseRestHandler.handleList(self, confInfo)\n  File "/opt/splunk/etc/system/bin/sc_rest.py", line 74, in handleList\n    ent = self.all()\n  File "/opt/splunk/etc/system/bin/sc_rest.py", line 221, in all\n    offset=self.posOffset)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities\n    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed\n    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 469, in simpleRequest\n    raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))\nSplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /servicesNS/-/-/data/models: _ssl.c:506: The handshake operation timed out',)\n

This issue was identified as deadlock bug in openssl.

[lpolo]

Issue address in release 6.1.4.
vulnerability is :CVE-2014-1912

[alexsayegh]

I had a similar problem when I tried to use SplunkDBConnect (dbx) app, i had similar errors (which were actually a missing mysql driver problem) and the jbridge was hanging all the time.

If you have DB inputs, check that you have the correct driver for DB engine:
http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Installdrivers

Also, in java.conf for dbx app you might want to add the bridge stanza:
[bridge]
addr = 127.0.0.1
port = XXX
threads = 10

This worked for me!

[lpolo]

Our problem is that splunkd stop responding due to a deadlock bug. Therefore, we are forced to restart the splunk service.

[lpolo]

I increased the ulimit to 65536. Let's see how it behaves.

[host]# ulimit -n
65536

[lpolo]

I have sent 2 diags to Splunk Tech support.
We have not received any work around or fix yet. I will update the notes once I have more information.

[sloshburch]

Anything result from this? I'm seeing the same issue.

[lpolo]

We are still being affected by this issue. We have captured pstacks. Splunk tech support identified a dead lock bug in openssl. However, the issue is not fixed in Splunk 6.1.2. Are you still being affected by this issue?