Why am I having trouble with TLS?

[NJ]

Hi everyone.

I have followed the documentation for setting up TLS for inter-Splunk communication with self-signed certificates.

I have a small test environment that has an SH, an Indexer and an UF. 

However, I get the following error:

03-15-2023 01:23:39.475 +0000 ERROR TcpInputProc [2605538 FwdDataReceiverThread] - Error encountered for connection from src=10.0.0.4:45088. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I have created the following certificates and keys based on the Splunk documentation: 

• myCertAuthCertificate.csr
• myCertAuthCertificate.pem
• myCertAuthCertificate.srl
• myCertAuthPrivateKey.key
• myServerCertificate.csr
• myServerCertificate.pem
• myServerPrivateKey.key
• mySplkCliCert.pem <- this is the concatenated file.

I copy the myCertAuthCertificate.pem and mySplkCliCert.pem files from the SH to the Indexer.

on the SH and Indexer, I edit the Server.conf to have the following:

[sslConfig]
sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem
serverCert = /opt/splunk/etc/auth/mycerts/mySplkCliCert.pem
sslPassword = *****

What am I doing wrong?

[michael_bates_1]

Thanks for that. 
In the inputs.conf on the indexer, can you please confirm that the SSL stanza is wrapper correctly.
Your config has the incorrect closing bracket.
[SSL}
rather than
[SSL]

Everything else looks good.
Another possible step is to revert to using the default certs created by Splunk when installing/running. 
This would eliminate any certificate issues such as access, wrong permissions, wrong order of certs, etc.

[NJ]

Thanks for spotting that!

However, after fixing the typo and restarting now I don't receive any logs from the forwarder or search head. 

[michael_bates_1]

The SSL settings in server.conf are to control the server used by splunkd and not what gets used in either the web ui or for tcp connections. It is entirely possible to different certificates for different parts of the Splunk solution.

[michael_bates_1]

Hi there. 
I strongly recommend that, at least to start with, do not do any client or server verification until the basic SSL connections are working.

Before heading down the rabbit hole of SSL, can I please confirm the following
1. outputs.conf on the sender (client)
     useSSL = true
     sslCertPath = path to client certificate (can be the self signed one)
     sslPassword
2. inputs.conf on the receiver (server)
     the input def should be splunktcp-ssl not splunktcp
     [SSL] stanza should be defined with the following
     serverCert = path to combined cert
     sslPassword
     requireClientCert = false

[triptraptresko]

TLDR: I was missing
1. outputs.conf on the sender (client)
useSSL = true

After following the documentation on how to enable ssl between forwarders and indexers, i got the error

ERROR TcpInputProc - Error encountered for connection from src=10.1.1.34:50772. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

In the documentation: https://docs.splunk.com/Documentation/Splunk/9.0.1/Security/ConfigureSplunkforwardingtousesignedcert...
, it specifies

[SSL]
requireClientCert = true

Which if you drop, will affect your outputs.conf -> useSSL.
It says if requireClientCert is defined, then useSSL will be true.
In my case, I mindlessly thought you could set requrieClientCert=false...

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Outputsconf
#----Secure Sockets Layer (SSL) Settings----

# To set up SSL on the forwarder, set the following setting/value pairs.
# If you want to use SSL for authentication, add a stanza for each receiver
# that must be certified.

useSSL = <true|false|legacy>
* Whether or not the forwarder uses SSL to connect to the receiver, or relies
  on the 'clientCert' setting to be active for SSL connections.
* You do not need to set 'clientCert' if 'requireClientCert' is set to
  "false" on the receiver.
* A value of "true" means the forwarder uses SSL to connect to the receiver.
* A value of "false" means the forwarder does not use SSL to connect to the
  receiver.
* The special value "legacy" means the forwarder uses the 'clientCert' property to
  determine whether or not to use SSL to connect.
* Default: legacy

[NJ]

Hi @michael_bates_1 

Thank for your comment,

Output.conf on my Search-Head:

[indexAndForward]
index = false

[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true
indexAndForward = false
autoLBFrequencyIntervalOnGroupFailure = -1
channelReapInterval = 60000
channelReapLowater = 10
channelTTL = 300000
connectionsPerTarget = 0
dnsResolutionInterval = 300
negotiateNewProtocol = true
polling_interval = 5
socksResolveDNS = false

[tcpout:my_search_peers]
server = 10.1.0.4:9997
useSSL = true
sslCertPath = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem
sslPassword = $7$AipeL1V0nT7oJ9t/qIGBOy0IZ6dBMzQtu8wATkwAwKwfwqd71K1gcGZBkF8=

outputs.conf on my UF

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.1.0.4:9997
useSSL = true
sslCertPath = /opt/splunkforwarder/etc/auth/mycerts/myServerCertificate.pem
sslPassword = $7$5+N0oNlLInzoFgzKRvrKrtFlhqrzpc9XwDx60n067DpXWBMYxuK5erjuURg=

[tcpout-server://10.1.0.4:9997]

Inputs.conf on the indexer

[splunktcp-ssl:9997]
disabled = 0

[SSL}
serverCert = /opt/splunk/etc/auth/mycerts/mySplkCliCert.pem
sslPassword = ********
requireClientCert = false