Getting Data In

Splunk Forwarder SSL error - "SSL23_GET_CLIENT_HELLO:unknown protocol"

grijhwani
Motivator

I just installed two new UFs (v5.0.9, identical to the indexer they are trying to communicate with). Despite picking up their configs from the deployment server and trying to direct their traffic to the correct indexer, tcpdump indicates some very short handshakes, and $SPLUNK_HOME/var/log/splunk/splunkd.log on each forwarder shows pairs of errors

INFO  TcpOutputProc - Connected to idx={indexerip}:9997
ERROR TcpOutputFd - Read error. Connection reset by peer

whilst the log on the indexer contains a stream of corresponding errors similar to

ERROR TcpInputProc - Error encountered for connection from src={forwarderip}:43479. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I already found Universal forwarders no longer sending data - SSL23 unknown which poses the question of whether the OpenSSL binaries have been relinked. They have not, and the binaries reported as embedded within Splunk are identical.

I'm looking for ideas of what gives. This is not a problem I have ever faced before after a simple UF install.

1 Solution

grijhwani
Motivator

It turns out it wasn't just the new forwarder, it was quite a few, and it was a simple mistake. The indexers are expecting compressed SSL traffic, and I had not set the SSL config.

View solution in original post

0 Karma

bbialek
Path Finder

I was getting this error when my inputs and outputs conf had encrypted sslPassword but I've forgotten to include the $SPLUNK_HOME/etc/auth/splunk.secret.

0 Karma

grijhwani
Motivator

It turns out it wasn't just the new forwarder, it was quite a few, and it was a simple mistake. The indexers are expecting compressed SSL traffic, and I had not set the SSL config.

0 Karma

DaClyde
Contributor

What was the solution here, had you just not set "compression = true" on the forwarders?

I just did that on my search head because I was getting the same error that my indexer wasn't receiving from the search head, but adding the compression setting to the outputs.conf on the SH didn't fix the problem. This was working for me on 6.2.1 before the 6.2.2 upgrade. After running the 6.2.2 upgrade, I get this error.

0 Karma

grijhwani
Motivator

I don't fully recall, but the UF's were configured by script, initially, and I think the ssl configuration was quite simply just missing in its totality.

~splunk/etc/system/local/server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true
0 Karma

wrangler2x
Motivator

What do you mean you had not set the SSL config? I am seeing this same thing. The funny thing is, the forwarder was working fine and all of a sudden stopped and I see the exact error you describe for it in my indexer's splunkd.log.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...