I have the same issue as documented in this posting. The answer makes sense. But I am not very comfortable with assuming that no one is going to attack port 8089.
I found a workaround for Apache 2 on StackExchange, which turns off SSL/TLS compression in openSSL. It works for me on Apache 2 but it does not work on splunkd.
I tried adding a line "export OPENSSLNODEFAULT_ZLIB=1" in /etc/init.d/splunk.
I also tried adding "OPENSSLNODEFAULT_ZLIB=1" in etc/splunk-launch.conf.
I am assuming that splunkd (Cherrypy) ultimately uses openSSL. If that is true, then the remedy should work. Could someone please respond to that?
Maybe I am doing it wrong. If that's the case, how do I inject an environment variable into the splunkd process?
Never mind. I found "OPENSSL_NO_DEFAULT_ZLIB=1" in the environment variable set of the running splunkd process. So the answer is that the workaround does not work for splunkd.
I am still curious, though, although splunkd uses its own SSL library, it still looks like openSSL (per lsof):
splunkd 10217 splunk mem REG 8,5 326064 270526 /opt/splunkforwarder/lib/libssl.so.0.9.8 splunkd 10218 splunk mem REG 8,5 326064 270526 /opt/splunkforwarder/lib/libssl.so.0.9.8
Splunk's libssl does not take the OPENSSL_NO_DEFAULT_ZLIB setting while the OS' copy does:
$ strings /opt/splunkforwarder/lib/libssl.so.0.9.8 | grep OPENSSL_NO_DEFAULT_ZLIB $ strings /usr/lib64/libssl.so.0.9.8 | grep OPENSSL_NO_DEFAULT_ZLIB OPENSSL_NO_DEFAULT_ZLIB
So it seems that Splunk could fix the issue without too much effort.
It looks like as of two different OpenSSL releases (1.0.1e and 0.9.8x) this
OPENSSL_NO_DEFAULT_ZLIB environment variable does not exist anywhere in the official openssl.org source code. Chasing its provenance down shows it to be a patch provided by someone at Novell that has not made it into the official openssl.org sources. The OS copy of OpenSSL would seem to have this patch applied to it, which is not uncommon for someone like Redhat (forinstance) to do.
Splunk could build their OpenSSL with the patch for this environment variable, or could build it without zLib support entirely, either one of which would help to mitigate this.
If they build OpenSSL without zLib support, this "should" cover both Splunkweb and Splunkd. You could request this via a support case / enhancement request.
But, one of the best mitigations for this type of issue is to defend that REST port. On most of our machines, the Splunkd REST port is entirely unreachable except for as absolutely necessary for things like license management and distributed search.
IMHO, Splunk needs to build their OpenSSL library with the patch.
I'll give three reasons:
Although port 8089 provides a REST service mostly used by other Splunk components, it is open to connection from any host by default. Extra care is needed to make this port only reachable to certain hosts. That could get tedious fast, especially when you have large number of forwarders.
This vulnerability exists on Splunk web port 8000 as well.
Further test shows that zLib compression is turned off if an SSL connection is initiated on interface 127.0.0.1 -- patching OpenSSL seems easier.
I can't disagree with you. If this is something you want, a support case asking for it would be the correct place to start. It would not hurt to make acquaintance with the right product managers within Splunk and let them know how important this is to you and your dollars you spend with them.