Splunk Forwarder SSL error - "SSL23_GET_CLIENT_HELLO:unknown protocol"

[mjmayer]

I'm attempting to setup splunk enterprise in a docker container using the official splunk image. I have been unsuccessful in getting ssl from the forwarders to the indexer configured. As far as I can tell, my certs are fine. I've tried enabling further debugging, but none of the errors seem to point me in the right direction.

indexer splunkd.log

    07-15-2017 07:41:52.160 +0000 INFO  TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
    07-15-2017 07:41:52.161 +0000 INFO  TcpInputProc - Creating raw Acceptor for IPv4 port 1514 with Non-SSL
    07-15-2017 07:41:52.161 +0000 INFO  TcpInputProc - Creating fwd data Acceptor for IPv4 port 9997 with SSL
    07-15-2017 07:43:45.404 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.101.21.34:36346. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
    07-15-2017 07:43:45.413 +0000 ERROR TcpInputProc - Error encountered for connection from src=10.101.21.34:36348. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

indexer inputs.conf

#[default]
#host = splunkenterprise

[splunktcp-ssl:9997]
disabled=0

[SSL]
serverCert = /opt/splunk/etc/auth/splunk.cert.chain.pem

indexer inputs.conf

[general]
serverName = splunkenterprise
pass4SymmKey = $1$liNoIVdm5xPP

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/intermediate-chain.cert.pem
sslPassword = $1$wW88fRIgrEHP

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

forwarder outputs.conf

[tcpout:group1]
server=splunk.mydomain.net:9997
disabled = 0

[tcpout:splunkssl]
sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem

forwarder server.conf

sslConfig]
sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
sslPassword = $1$rCb/2hSpZ34D

[general]
pass4SymmKey = $1$+2qrhlHvLCwD

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

/splunk cmd openssl s_client -connect splunk.mydomain.net:9997

New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: zlib compression
Expansion: zlib compression
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-GCM-SHA384
    Session-ID: 443457B09EEBEE91F8B72DE5132E970CCDBD14D96A1BF5BE02FE34ED6EA631D9
    Session-ID-ctx:   
    Master-Key: B1D47C8F8EDD71D957E7BBC78B946B3FCFC4B6FB5B5527C3E16C6ADDE7C1DF7A6B950E8B2DC148EFCA4A70D88BC6035E
    Key-Arg   : None  
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - d6 cc 25 7b f8 a9 96 eb-70 16 9c ce 01 99 7f 0e   ..%{....p.......
    0010 - fd 7d 06 ec 4f cc 9f 63-27 00 b4 c2 19 b7 fc c0   .}..O..c'.......
    0020 - e9 c5 0c 9c 2c 21 ed df-28 34 bd 4f 00 68 87 d2   ....,!..(4.O.h..
    0030 - b2 cc bb 2a ff 32 51 26-dc 4f 7d d8 36 94 ea 96   ...*.2Q&.O}.6...
    0040 - 74 d6 16 16 35 1e 3e c9-84 6c f0 c1 4a 3a 8d 7c   t...5.>..l..J:.|
    0050 - 1b 8d 67 bf 77 95 a2 1a-4a 4b df 2e 8a 12 4d fb   ..g.w...JK....M.
    0060 - 05 e5 39 74 28 2a 5a 35-39 70 57 09 f0 5c ac aa   ..9t(*Z59pW..\..
    0070 - 31 82 b1 42 24 8f 80 6e-6b a9 97 c5 31 2d e9 6a   1..B$..nk...1-.j
    0080 - c1 03 25 c0 27 53 90 b0-7d 2e 1a 1a a5 24 73 6c   ..%.'S..}....$sl
    0090 - 89 4e 3d f7 2d f7 dd 1c-ce 0c 65 36 f6 27 55 11   .N=.-.....e6.'U.

    Compression: 1 (zlib compression)
    Start Time: 1500105568
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)

[jkat54 · Accepted solution]

the server and ssl settings should be in the same tcpout stanza.

[tcpout:group1]
server=splunk.mydomain.net:9997
clientCert = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
sslPassword= clientCertPasswordIfApplicable

[mjmayer]

Switching the outputs stanza as you advised to

    [tcpout:group1]
    server=splunk.mydomain.net:9997
    disabled = 0
    sslRootCAPath = /etc/pki/ca-trust/source/anchors/intermediate-chain.cert.pem
    clientCert = /opt/splunkforwarder/etc/auth/splunk.mydomain.net.cert.pair.pem

worked. The client is now logging

7-15-2017 18:39:55.889 +0000 INFO  TcpOutputProc - Connected to idx=10.101.21.34:9997

The indexer is logging

07-15-2017 18:34:59.115 +0000 DEBUG TcpInputConfig - connection_host=ip for 10.101.21.34

It does seem odd that the instructions show that it us supposed to be configured differently

[tcpout:group1]
server=10.1.1.197:9997
disabled = 0

[tcpout:splunkssl]
useClientSSLCompression = <true> Disabling tls compression can cause bandwidth issues.
sslPassword = The password for the CAcert
sslCommonNameToCheck = (Optional) <commonName1>, <commonName2>, ... sslVerifyServerCert must be enabled to use common name checking. Defaults to no common name checking. 
sslAltNameToCheck = (Optional) <alternateName1>, <alternateName2>, ... sslVerifyServerCert must be enabled to use common name checking. Defaults to no common name checking. 
sslVerifyServerCert =  Defaults to false. If true, you must make sure that the server you are connecting to can be authenticated to. When enabled, the common name and the alternate name of the server are checked.
cipherSuite = (Optional) Splunk uses any specified cipher string for the input processors. If not set, Splunk uses the default cipher string provided by OpenSSL.

[woodcock]

Leave a Comment at the bottom of the docs page telling them that the information is outdated/incorrect.

[jkat54]

Since this "sslRootCAPath" is deprecated, you don't need it. I would also be interested in seeing which instructions specified those settings as I don't see it in outputs.conf documentation.

[mjmayer]

http://docs.splunk.com/Documentation/Splunk/6.5.3/Security/ConfigureSplunkforwardingtousesignedcerti...

[jkat54]

I submitted feedback. Thanks!

[jkat54]

If you can share a link,
I can ask the author to revise.

[cpetterborg]

Did you recently upgrade to 6.6.x?

Do you have some older UFs?

Are you using SSL on the UFs to communicate?

If so, it sounds like the V2 vs V3 SSL problem that you can find several answers for here.

[mjmayer]

This is a new install of the indexer and forwarder. In my original configuration I was running 6.5.3 on the indexer and 6.6.0 for the forwarder.

You did get me thinking. I was running indexer 6.5.3 and a 6.6.0 forwarder. I did bring the forwarder down to version 6.5.3, but I'm seeing the same errors.