HEC only accepts TLS 1.2 no matter what sslVersions

[yuanliu]

My data source can't seem to negotiate TLS v1.2.  So, I am trying to "downgrade" HEC.  But no matter how I change inputs.conf, only TLS 1.2 is supported on port 8080.

In fact, default sslVersions for splunk_httpinput app is already *:

$ cat etc/apps/splunk_httpinput/default/inputs.conf
[http]
disabled=1
port=8088
enableSSL=1
dedicatedIoThreads=2
maxThreads = 0
maxSockets = 0
useDeploymentServer=0
# ssl settings are similar to mgmt server
sslVersions=*,-ssl2
allowSslCompression=true
allowSslRenegotiation=true
ackIdleCleanup=true

openssl s_client can only negotiate within TLS 1.2, nothing lower.  If I use, say -tls1_1, splunkd.log shows "error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher", the same error my data source triggers.  Is there some way to "downgrade"?

The data source in question is Puppet Inc's splunk_hec module used by Puppet Report Viewer (Splunk base app 4413 ).  I am testing it with Puppet Server 2.7.0. (Splunk is 8.2.0.) My colleague suspects that the Jruby version (ruby 1.9) may be too old to support TLS 1.2. (I can invoke splunk_hec report in native Ruby 2.0 successfully.)

Update: JRuby version is probably the problem, although it does support TLS 1.2; the problem is (still) in cipher suites mismatch.  I used tcpdump and wireshark to analyze TLS exchange.  Puppet server sends the following:

Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 223
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 219
            Version: TLS 1.2 (0x0303)
            Random: c1221d62f8911dc203ac02cf12c7cf7a71093cd5141a0f56e7bad2429d4e1095
            Session ID Length: 0
            Cipher Suites Length: 12
            Cipher Suites (6 suites)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)

Even adding extended ciphers illustrated in default inputs.conf, i.e.,

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA

they still cannot match.

I am unfamiliar with how Splunk represents these suites.  Is there a supported cipher suite that can match one of those sent by splunk_hec?

[yuanliu]

---

@yuanliu wrote:

My data source can't seem to negotiate TLS v1.2.  So, I am trying to "downgrade" HEC.  But no matter how I change inputs.conf, only TLS 1.2 is supported on port 8080.

...

...  Is there some way to "downgrade"?

I'm going to revert to the original scope and close this one: As it turns out, in order to make splunk_httpinput app (HEC interface, default port 8088) accept any protocol, that protocol must have already been accepted in web.conf.

By default, web.conf only accepts TLSv1.2.  Therefore splunk_httpinput can only accept TLSv1.2 no matter what etc/apps/splunk_httpintput/local/inputs.conf says.  Once I add other protocols to etc/system/local/web.conf, I can change HEC behavior by updating etc/apps/splunk_httpintput/local/inputs.conf.

In my opinion, splunk_httpinput app is orthogonal to splunkd's Web interface.  So, the intertwinement between the two configurations should be a bug.

Additionally, [SSL] stanza in etc/system/local/inputs.conf seems to have no effect on any interface.  Not sure if that should be a bug, but at least that is not what documentation and comments in etc/system/default/inputs.conf imply.  If it is not a product bug, it is a documentation bug.

[yuanliu · Accepted solution]

---

@yuanliu wrote:

... As it turns out, in order to make splunk_httpinput app (HEC interface, default port 8088) accept any protocol, that protocol must have already been accepted in web.conf.

After even more tests, it turns out that web.conf does not affect HEC, after all.  Nor does $SPLUNK_HOME/etc/system/local/inputs.conf.  My mistake was to mix up results from Puppet server with those from openssl.  To be sure, HEC by default only support TLSv1.2 despite sslVersions=*, because only TLSv1.2 supports the default cipherSuite (which is not exemplified in $SPLUNK_HOME/etc/apps/splunk_httpinput/default/inputs.conf).

In order for HEC to support TLSv1.1 (as well as TLSv1.2), therefore, I have to copy default cipherSuite values from $SPLUNK_HOME/etc/system/default/web.conf (or server.conf), then add suitable TLSv1.1 ciphers (such as AES256_SHA) into $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf.  No need to alter web.conf.)

[yuanliu]

To look at JRuby's offer again: they are all from SSLv3 - not applicable for TLSv1.2.

$ openssl ciphers -V |grep 0x00,0x3[23589]
          0x00,0x39 - DHE-RSA-AES256-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA1
          0x00,0x38 - DHE-DSS-AES256-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(256)  Mac=SHA1
          0x00,0x35 - AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
          0x00,0x33 - DHE-RSA-AES128-SHA      SSLv3 Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA1
          0x00,0x32 - DHE-DSS-AES128-SHA      SSLv3 Kx=DH       Au=DSS  Enc=AES(128)  Mac=SHA1

So, this is an impasse: On one hand, splunk_httpinput doesn't allow any version but TLS 1.2.  On the other, JRuby 1.9 offers nothing but SSL3 cipher suites.

Is there any way to change either behavior?  I tried to explicitly set sslVersion in etc/apps/splunk_httpinput/local/inputs.conf and every time, only TLS 1.2 is allowed.  I don't know how to influence puppetserver 2.1 to make JRuby connect method offer TLS 1.2. (TLS 1.2 isn't all that new, and SSLv3 is extremely ancient.  This client behavior is very strange even for an old version of Ruby.)

[yuanliu]

---

I am unfamiliar with how Splunk represents these suites.  Is there a supported cipher suite that can match one of those sent by splunk_hec?

---

Well, this observation gives me an idea: Use *

cipherSuite = *

This is not ideal for obvious reasons.  But at least splunk_hec is no longer complaining about no common cipher. 

[yuanliu]

---

cipherSuite = *

Unfortunately, turns out splunkd - or the splunk_httpinput app portion of it, doesn't like *.  Any attempt to connect to HEC port simply times out. (Which I reported seprately in https://community.splunk.com/t5/All-Apps-and-Add-ons/Puppet-Could-not-send-report-to-Splunk-executio... not realizing the causal connection.)

So, back to square one: How can I configure splunk_httpinput to match JRuby 1.9's suites? Or allow, say, TLS 1.1.

I have already added Windows Vista support as illustrated in comments, i.e., 

ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA

I would think that ECDHE-RSA-AES128-SHA, for example, would match for TLS_DHE_DSS_WITH_AES_128_CBC_SHA

[venkatasri]

Hi @yuanliu 

Have you tried lowering sslVersion in server.conf to tls1.1,  if SSL management port enabled with SSL?

[yuanliu]

This is not management port, but HEC port 8088.  The corresponding config is under apps/splunk_httpinput/local/inputs.conf; the default sslVersion is already *.  I have tried to lower sslVersion to tls as suggested in system/local/input.conf to no effect. (In fact, that one didn't affect management port 8089, either.)  If I change system/local/web.conf, on the other hand, I can observe port 443 allowing lower grade protocols.  I believe that management port is controlled by system/local/server.conf although I didn't explicitly test.