Getting Data In

Splunk Forwarder SSL error - "SSL23_GET_CLIENT_HELLO:unknown protocol"

grijhwani
Motivator

I just installed two new UFs (v5.0.9, identical to the indexer they are trying to communicate with). Despite picking up their configs from the deployment server and trying to direct their traffic to the correct indexer, tcpdump indicates some very short handshakes, and $SPLUNK_HOME/var/log/splunk/splunkd.log on each forwarder shows pairs of errors

INFO  TcpOutputProc - Connected to idx={indexerip}:9997
ERROR TcpOutputFd - Read error. Connection reset by peer

whilst the log on the indexer contains a stream of corresponding errors similar to

ERROR TcpInputProc - Error encountered for connection from src={forwarderip}:43479. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I already found Universal forwarders no longer sending data - SSL23 unknown which poses the question of whether the OpenSSL binaries have been relinked. They have not, and the binaries reported as embedded within Splunk are identical.

I'm looking for ideas of what gives. This is not a problem I have ever faced before after a simple UF install.

1 Solution

grijhwani
Motivator

It turns out it wasn't just the new forwarder, it was quite a few, and it was a simple mistake. The indexers are expecting compressed SSL traffic, and I had not set the SSL config.

View solution in original post

0 Karma

bbialek
Path Finder

I was getting this error when my inputs and outputs conf had encrypted sslPassword but I've forgotten to include the $SPLUNK_HOME/etc/auth/splunk.secret.

0 Karma

grijhwani
Motivator

It turns out it wasn't just the new forwarder, it was quite a few, and it was a simple mistake. The indexers are expecting compressed SSL traffic, and I had not set the SSL config.

0 Karma

DaClyde
Contributor

What was the solution here, had you just not set "compression = true" on the forwarders?

I just did that on my search head because I was getting the same error that my indexer wasn't receiving from the search head, but adding the compression setting to the outputs.conf on the SH didn't fix the problem. This was working for me on 6.2.1 before the 6.2.2 upgrade. After running the 6.2.2 upgrade, I get this error.

0 Karma

grijhwani
Motivator

I don't fully recall, but the UF's were configured by script, initially, and I think the ssl configuration was quite simply just missing in its totality.

~splunk/etc/system/local/server.conf

[sslConfig]
enableSplunkdSSL = true
useClientSSLCompression = true
useSplunkdClientSSLCompression = true
0 Karma

wrangler2x
Motivator

What do you mean you had not set the SSL config? I am seeing this same thing. The funny thing is, the forwarder was working fine and all of a sudden stopped and I see the exact error you describe for it in my indexer's splunkd.log.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...