Monitoring Splunk

splunkd stop responding -> ERROR AdminManager

lpolo
Motivator

Splunkd stop responding after the event presented below.

Splunk Tech. Support filed a bug against Splunk Enterprise version 6.0.1.

02-12-2014 13:59:01.768 +0000 ERROR AdminManager - Stack trace from python handler:\nTraceback (most recent call last):\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 70, in init\n    hand.execute(info)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 527, in execute\n    if self.requestedAction == ACTION_LIST:     self.handleList(confInfo)\n  File "/opt/splunk/etc/system/bin/DataModelAccelerationHandler.py", line 20, in handleList\n    sc_rest.BaseRestHandler.handleList(self, confInfo)\n  File "/opt/splunk/etc/system/bin/sc_rest.py", line 74, in handleList\n    ent = self.all()\n  File "/opt/splunk/etc/system/bin/sc_rest.py", line 221, in all\n    offset=self.posOffset)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 129, in getEntities\n    atomFeed = _getEntitiesAtomFeed(entityPath, namespace, owner, search, count, offset, sort_key, sort_dir, sessionKey, uri, hostPath, **kwargs)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 222, in _getEntitiesAtomFeed\n    serverResponse, serverContent = rest.simpleRequest(uri, getargs=kwargs, sessionKey=sessionKey, raiseAllErrors=True)\n  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 469, in simpleRequest\n    raise splunk.SplunkdConnectionException, 'Error connecting to %s: %s' % (path, str(e))\nSplunkdConnectionException: Splunkd daemon is not responding: ('Error connecting to /servicesNS/-/-/data/models: _ssl.c:506: The handshake operation timed out',)\n

This issue was identified as deadlock bug in openssl.

Tags (2)

lpolo
Motivator

Issue address in release 6.1.4.
vulnerability is :CVE-2014-1912

0 Karma

alexsayegh
Explorer

I had a similar problem when I tried to use SplunkDBConnect (dbx) app, i had similar errors (which were actually a missing mysql driver problem) and the jbridge was hanging all the time.

If you have DB inputs, check that you have the correct driver for DB engine:
http://docs.splunk.com/Documentation/DBX/latest/DeployDBX/Installdrivers

Also, in java.conf for dbx app you might want to add the bridge stanza:
[bridge]
addr = 127.0.0.1
port = XXX
threads = 10

This worked for me!

0 Karma

lpolo
Motivator

Our problem is that splunkd stop responding due to a deadlock bug. Therefore, we are forced to restart the splunk service.

0 Karma

lpolo
Motivator

I increased the ulimit to 65536. Let's see how it behaves.

[host]# ulimit -n
65536

0 Karma

lpolo
Motivator

I have sent 2 diags to Splunk Tech support.
We have not received any work around or fix yet. I will update the notes once I have more information.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Anything result from this? I'm seeing the same issue.

0 Karma

lpolo
Motivator

We are still being affected by this issue. We have captured pstacks. Splunk tech support identified a dead lock bug in openssl. However, the issue is not fixed in Splunk 6.1.2. Are you still being affected by this issue?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...