Splunk Forwarder SSL error - "SSL23_GET_CLIENT_HELLO:unknown protocol"

[grijhwani]

I just installed two new UFs (v5.0.9, identical to the indexer they are trying to communicate with). Despite picking up their configs from the deployment server and trying to direct their traffic to the correct indexer, tcpdump indicates some very short handshakes, and $SPLUNK_HOME/var/log/splunk/splunkd.log on each forwarder shows pairs of errors

INFO  TcpOutputProc - Connected to idx={indexerip}:9997
ERROR TcpOutputFd - Read error. Connection reset by peer

whilst the log on the indexer contains a stream of corresponding errors similar to

ERROR TcpInputProc - Error encountered for connection from src={forwarderip}:43479. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I already found Universal forwarders no longer sending data - SSL23 unknown which poses the question of whether the OpenSSL binaries have been relinked. They have not, and the binaries reported as embedded within Splunk are identical.

I'm looking for ideas of what gives. This is not a problem I have ever faced before after a simple UF install.

[bbialek]

I was getting this error when my inputs and outputs conf had encrypted sslPassword but I've forgotten to include the $SPLUNK_HOME/etc/auth/splunk.secret.

[grijhwani · Accepted solution]

It turns out it wasn't just the new forwarder, it was quite a few, and it was a simple mistake. The indexers are expecting compressed SSL traffic, and I had not set the SSL config.

[DaClyde]

What was the solution here, had you just not set "compression = true" on the forwarders?

I just did that on my search head because I was getting the same error that my indexer wasn't receiving from the search head, but adding the compression setting to the outputs.conf on the SH didn't fix the problem. This was working for me on 6.2.1 before the 6.2.2 upgrade. After running the 6.2.2 upgrade, I get this error.

[grijhwani]

I don't fully recall, but the UF's were configured by script, initially, and I think the ssl configuration was quite simply just missing in its totality.

~splunk/etc/system/local/server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true

[wrangler2x]

What do you mean you had not set the SSL config? I am seeing this same thing. The funny thing is, the forwarder was working fine and all of a sudden stopped and I see the exact error you describe for it in my indexer's splunkd.log.