Indexer/UF SSL: requireClientCert and SSL3_GET_RECORD:wrong version number (7.3.2)

[ohbuckeyeio]

Hello,

I have been working to enable SSL between a UF and an indexer and am not sure if I follow the usage of the requireClientCert option. It seems to me the purpose of this option is disabling a two-way handshake between the forwarder and indexer, but the behavior I am seeing is counter to that thought.

If I do not point the forwarder's output.conf to a clientCert and sslPassword, I receive this error in the indexer log:

04-27-2020 19:48:52.747 +0000 ERROR TcpInputProc - Error encountered for connection from src=my_fwdr_ip:38694. error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

That's a pretty generic error, but in most cases it means there was a handshake issue between a client and server. Shouldn't the requireClientCert=false negate the necessity for the forwarder to present a cert back to the indexer? Is this a bug?

Below are my .confs

inputs.conf on indexer

[default]
host = myhost.mycodomain

[splunktcp-ssl:9997]
disabled = 0

[SSL]
serverCert = /opt/splunk/etc/auth/myco_certs/mychain.pem
sslPassword = <redacted>
requireClientCert = false

outputs.conf on UF

[tcpout]
disabled = false
defaultGroup = splkgroup1

[tcpout:splkgroup1]
server = 123.456.123.456:9997
disabled = 0
sslCommonNameToCheck = myco.com
sslVerifyServerCert = true

[harsmarvania57 · Accepted solution]

Hi,

On Indexer & UF, you are missing CAcert configuration
Indexer server.conf

sslRootCAPath = <Absolute path to the CA certificate, for example, the default value is 
$SPLUNK_HOME/etc/auth/cacert.pem>

UF server.conf

[sslConfig]
sslRootCAPath = <absolute path to the CA cert, for example, the default value 
is $SPLUNK_HOME/etc/auth/cacert.pem>

When you set requireClientCert=false, in this scenario handshake will still happen and if certificate is not signed by same root CA then handshake will fail. requireClientCert is mainly used with sslCommonNameToCheck so that Indexer will accept traffic from those forwarders whose certificate will match with Indexer sslCommonNameToCheck

[ohbuckeyeio]

Thank you. So requireClientCert=false is not the same as enabling a one-way handshake? I do have a cert for the forwarder. I am just trying to understand the usage of requireClientCert.

Also, I do have the sslRootCAPath configured. Everything works when I have requireClientCert=true on the indexer and clientCert and sslPassword on the UF. It's when I remove those and set requireClientCert=false that it breaks.

[harsmarvania57]

If you do not want to do handshake then you can use useSSL = true in outputs.conf on UF and in this case you do not need clientCert. (This is like you are browsing website and you do not have root CA in your browser)

useSSL = <true|false|legacy>
* Whether or not the forwarder uses SSL to connect to the receiver, or relies
  on the 'clientCert' setting to be active for SSL connections.
* You do not need to set 'clientCert' if 'requireClientCert' is set to
  "false" on the receiver.
* If set to "true", then the forwarder uses SSL to connect to the receiver.
* If set to "false", then the forwarder does not use SSL to connect to the
  receiver.
* If set to "legacy", then the forwarder uses the 'clientCert' property to
  determine whether or not to use SSL to connect.
* Default: legacy

When you set requireClientCert=false on Indexer, have you tried to configure sslVerifyServerCert = false in outputs.conf on UF & check?

[ohbuckeyeio]

I did not try sslVerifyServerCert but I also had not found the useSLL option. I believe that option answers my question, and after playing around with the requireClientCert after your first post, this is all much more understandable. Thank you!

[PavelP]

Hello @ohbuckeyeio,

this error indicates the indexer expects TLS but get something different, most probably plain TCP connection. Double check you set all correct on the forwarder (for example clientCert etc), check this : https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/ConfigureSplunkforwardingtousesignedcert...

Let me know how if it worked

[ohbuckeyeio]

I have been through all of that. The question I am asking is why all of that is needed if requireClientCert=false on the indexer. To me, this says the indexer is not expecting the forwarder to have a cert.