Puppet Could not send report to Splunk: execution expired

[yuanliu]

The following error is captured in puppetserver.log (no error in splunkd.log):

[puppetserver] Puppet Could not send report to Splunk: execution expired
["org/jruby/ext/openssl/SSLSocket.java:215:in `connect'", "/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'", "org/jruby/ext/timeout/Timeout.java:115:in `timeout'", "/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:800:in `connect'", "/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:756:in `do_start'", "/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:745:in `start'", "/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/net/http.rb:1293:in `request'", "/etc/puppetlabs/code/environments/production/modules/splunk_hec/lib/puppet/util/splunk_hec.rb:57:in `submit_request'", "/etc/puppetlabs/code/environments/production/modules/splunk_hec/lib/puppet/reports/splunk_hec.rb:112:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:53:in `processors'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:176:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:306:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'", "file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'", "Puppet$$Server$$Master_576124986.gen:13:in `handleRequest'", "request_handler_core.clj:273:in `invoke'", "jruby_request.clj:46:in `invoke'", "jruby_request.clj:31:in `invoke'", "request_handler_service.clj:34:in `handle_request'", "request_handler.clj:3:in `invoke'", "request_handler.clj:3:in `invoke'", "core.clj:2515:in `invoke'", "core.clj:211:in `invoke'", "core.clj:45:in `invoke'", "core.clj:343:in `invoke'", "core.clj:51:in `invoke'", "ringutils.clj:83:in `invoke'", "master_core.clj:430:in `invoke'", "ring.clj:21:in `invoke'", "ring.clj:12:in `invoke'", "comidi.clj:249:in `invoke'", "jetty9_core.clj:424:in `invoke'", "normalized_uri_helpers.clj:80:in `invoke'"]

From the puppet server's shell, puppet apply --report=splunk_hec is able to send report with no error.

(Puppet Inc's splunk_hec reporter is used by Puppet Report Viewer (Splunk base app 4413 ).  My environment is puppetserver 2.7.0; Splunk is 8.2.0.)

[yuanliu]

This turns out to be the other side of the coin in my effort to fix https://community.splunk.com/t5/Getting-Data-In/HEC-ssl3-get-client-hello-no-shared-cipher/m-p/56133....  Basically Puppet server 2.7.0 can only offer older cipher suites in TLS 1.2, and I don't know how to configure etc/app/splunk_httpsinput/local/inputs.conf with a list that matches splunk_hec's offer, or how to make HEC (port 8088) downgrade to TLS 1.1.

When I try

cipherSuite = *

Splunkd starts, and listens to all ports including 8088, spills no error.  But any client that attempts connect simply hangs.  splunk_hec has a timeout parameter,  hence "expired".  What do I need to do in order to match cipher suites on the two sides?

Update: Whereas I still haven't found a method to match ciphers of splunk_hec reporter in Puppet and splunk_httpintput app in Splunk (HEC interface), I eventually found how to "downgrade" HEC's ssl_version as indicated in the other investigation.

In the original scope of the question, the answer is: splunk_httpinput app freezes when input cipherSuite = *.  This is a bug in my books.

[yuanliu · Accepted solution]

After confirming that Splunkd's mishandled cipherSuite=*, I finally found a workaround to allow puppetserver 2.7.0 to talk to Splunk 8.2.  The short answer is to add one or more ciphers that the outdated JRuby actually offers to cipherSuite in $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf. (Technically, adding all ciphers that TLSv1.1 supports is easier.  But doing so unnecessarily increases vulnerability of this downgrade.)

In practice, this requires some planning.  Here is my procedure.

• Capture TCP traffic on HEC when puppetserver attempts to connect.  For splunk_hec, every agent execution invokes a client connection.
• Use Wireshark to decode TLS on the HEC port (8088).  Because the client fails, you will only see client hello and a server warning.  This will reveal cipher offered by puppetserver.  My server offers 11:

Cipher Suites (11 suites)
    Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
    Cipher Suite: TLS_DHE_RSA_WITH_DES_CBC_SHA (0x0015)
    Cipher Suite: TLS_DH_anon_WITH_DES_CBC_SHA (0x001a)
    Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)
    Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)
    Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
    Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002)
    Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c)
    Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
    Cipher Suite: TLS_ECDH_anon_WITH_RC4_128_SHA (0xc016)
    Cipher Suite: TLS_DH_anon_WITH_RC4_128_MD5 (0x0018)​

(Although I am still uncertain what affects puppetserver's offers and despite Puppet bug SERVER-2036 preventing configuration of http-client ciphers, there is indication that some installations may offer a different list; e.g., https://ask.puppet.com/question/33316/puppet-https-connection-using-latest-tls-version-and-cipher-su... implies that AES256_SHA (TLS_RSA_WITH_AES_256_CBC_SHA) is offered but that is not my case.) Note Wireshark lists cipher suites in IANA notation which is also used in Puppet configuration files even though Ruby and JRuby are compiled against openssl.  We shall use the hexadecimal representation (0x0018, etc.) to match what Splunk can support.

• On Splunk server, use 

openssl ciphers -V​​

to list ciphers that Splunk can support.  The list will be in openssl notation which is also used in Splunk configuration files.  This is why we need -V to reveal ciphers' hexadecimal representation for matching with Puppet server's http-client.  Partial list in openssl 1.0.2k-fips is as follows:

          0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD
          0xC0,0x2C - ECDHE-ECDSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(256) Mac=AEAD
          0xC0,0x28 - ECDHE-RSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA384
          0xC0,0x24 - ECDHE-ECDSA-AES256-SHA384 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA384
          0xC0,0x14 - ECDHE-RSA-AES256-SHA    SSLv3 Kx=ECDH     Au=RSA  Enc=AES(256)  Mac=SHA1
          0xC0,0x0A - ECDHE-ECDSA-AES256-SHA  SSLv3 Kx=ECDH     Au=ECDSA Enc=AES(256)  Mac=SHA1
...
          0x00,0x3D - AES256-SHA256           TLSv1.2 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA256
          0x00,0x35 - AES256-SHA              SSLv3 Kx=RSA      Au=RSA  Enc=AES(256)  Mac=SHA1
...
          0xC0,0x11 - ECDHE-RSA-RC4-SHA       SSLv3 Kx=ECDH     Au=RSA  Enc=RC4(128)  Mac=SHA1
          0xC0,0x07 - ECDHE-ECDSA-RC4-SHA     SSLv3 Kx=ECDH     Au=ECDSA Enc=RC4(128)  Mac=SHA1
          0xC0,0x0C - ECDH-RSA-RC4-SHA        SSLv3 Kx=ECDH/RSA Au=ECDH Enc=RC4(128)  Mac=SHA1
...

• Compare hex codes between Wireshark output and openssl listing, then add one or more matching protocols into cipherSuite in $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf.  Note Wireshark uses 2-byte, lower case Hex numeral whereas openssl uses two 1-byte, upper case numerals.  For example, if I choose 0xc011 from puppetserver's offers, I can use

$ openssl ciphers -V |fgrep 0xC0,0x11​
          0xC0,0x11 - ECDHE-RSA-RC4-SHA       SSLv3 Kx=ECDH     Au=RSA  Enc=RC4(128)  Mac=SHA1

to determine that ECDHE-RSA-RC4-SHA is the corresponding openssl notation. Do not forget to also copy cipherSuite from $SPLUNK_HOME/etc/system/default/web.conf so TLSv1.2 is also supported.

In my case, I notice that 0xc011 => 0xC0,0x11 exists in both lists: ECDHE-RSA-RC4-SHA in openssl and TLS_ECDHE_RSA_WITH_RC4_128_SHA in IANA.  So, my entry looks like

cipherSuite = ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256​

(All ciphers after the first one are copied from web.conf.)

With this, splunk_hec is able to send data to Splunk without changing Puppet server.

[yuanliu]

---

https://ask.puppet.com/question/33316/puppet-https-connection-using-latest-tls-version-and-cipher-su... implies that AES256_SHA (TLS_RSA_WITH_AES_256_CBC_SHA) is offered but that is not my case.

---

Oddly enough, another instance of puppetserver 2.7.0, with the exact same openjdk, openssl, ruby, and JRuby, offers a totally different set of ciphers that does include AES256_SHA (0x0035).

Cipher Suites (6 suites)
    Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
    Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
    Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
    Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)

I also find that even though 0x0039 (TLS_DHE_RSA_WITH_AES_256_CBC_SHA/DHE-RSA-AES256-SHA) is supported in openssl on the Splunk server, splunk_httpinput still won't negotiate on that.