topic Re: Why am I having trouble with TLS? in Getting Data In

Why am I having trouble with TLS?

NJ — Wed, 15 Mar 2023 18:29:49 GMT

Hi everyone.

I have followed the documentation for setting up TLS for inter-Splunk communication with self-signed certificates.

I have a small test environment that has an SH, an Indexer and an UF.

However, I get the following error:

03-15-2023 01:23:39.475 +0000 ERROR TcpInputProc [2605538 FwdDataReceiverThread] - Error encountered for connection from src=10.0.0.4:45088. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

I have created the following certificates and keys based on the Splunk documentation:

myCertAuthCertificate.csr
myCertAuthCertificate.pem
myCertAuthCertificate.srl
myCertAuthPrivateKey.key
myServerCertificate.csr
myServerCertificate.pem
myServerPrivateKey.key
mySplkCliCert.pem <- this is the concatenated file.

I copy the myCertAuthCertificate.pem and mySplkCliCert.pem files from the SH to the Indexer.

on the SH and Indexer, I edit the Server.conf to have the following:

[sslConfig] sslRootCAPath = /opt/splunk/etc/auth/mycerts/myCertAuthCertificate.pem serverCert = /opt/splunk/etc/auth/mycerts/mySplkCliCert.pem sslPassword = *****

What am I doing wrong?

Re: Trouble with TLS

michael_bates_1 — Wed, 15 Mar 2023 06:21:35 GMT

Hi there.
I strongly recommend that, at least to start with, do not do any client or server verification until the basic SSL connections are working.

Before heading down the rabbit hole of SSL, can I please confirm the following
1. outputs.conf on the sender (client)
useSSL = true
sslCertPath = path to client certificate (can be the self signed one)
sslPassword
2. inputs.conf on the receiver (server)
the input def should be splunktcp-ssl not splunktcp
[SSL] stanza should be defined with the following
serverCert = path to combined cert
sslPassword
requireClientCert = false

Re: Trouble with TLS

michael_bates_1 — Wed, 15 Mar 2023 06:23:35 GMT

The SSL settings in server.conf are to control the server used by splunkd and not what gets used in either the web ui or for tcp connections. It is entirely possible to different certificates for different parts of the Splunk solution.

Re: Trouble with TLS

NJ — Wed, 15 Mar 2023 23:03:04 GMT

Hi @michael_bates_1

Thank for your comment,

Output.conf on my Search-Head:

[indexAndForward] index = false [tcpout] defaultGroup = my_search_peers forwardedindex.filter.disable = true indexAndForward = false autoLBFrequencyIntervalOnGroupFailure = -1 channelReapInterval = 60000 channelReapLowater = 10 channelTTL = 300000 connectionsPerTarget = 0 dnsResolutionInterval = 300 negotiateNewProtocol = true polling_interval = 5 socksResolveDNS = false [tcpout:my_search_peers] server = 10.1.0.4:9997 useSSL = true sslCertPath = /opt/splunk/etc/auth/mycerts/myServerCertificate.pem sslPassword = $7$AipeL1V0nT7oJ9t/qIGBOy0IZ6dBMzQtu8wATkwAwKwfwqd71K1gcGZBkF8=

outputs.conf on my UF

[tcpout] defaultGroup = default-autolb-group [tcpout:default-autolb-group] server = 10.1.0.4:9997 useSSL = true sslCertPath = /opt/splunkforwarder/etc/auth/mycerts/myServerCertificate.pem sslPassword = $7$5+N0oNlLInzoFgzKRvrKrtFlhqrzpc9XwDx60n067DpXWBMYxuK5erjuURg= [tcpout-server://10.1.0.4:9997]

Inputs.conf on the indexer

[splunktcp-ssl:9997] disabled = 0 [SSL} serverCert = /opt/splunk/etc/auth/mycerts/mySplkCliCert.pem sslPassword = ******** requireClientCert = false

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 00:39:11 GMT

Thanks for that.
In the inputs.conf on the indexer, can you please confirm that the SSL stanza is wrapper correctly.
Your config has the incorrect closing bracket.
[SSL}
rather than
[SSL]

Everything else looks good.
Another possible step is to revert to using the default certs created by Splunk when installing/running.
This would eliminate any certificate issues such as access, wrong permissions, wrong order of certs, etc.

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 00:49:51 GMT

Thanks for spotting that!

However, after fixing the typo and restarting now I don't receive any logs from the forwarder or search head.

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 00:56:56 GMT

Are you able to revert all elements to use the default certificate
/etc/auth/server.pem

You can also check the internal logs (index=_internal) and look for x509
This will normally on appear after a reboot.

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 01:00:36 GMT

So i should change the output.conf to use the /opt/splunk/etc/auth/server.pem?

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 01:03:18 GMT

Change both ends just to be sure.

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 01:15:45 GMT

Should inputs.conf have serverCert = /opt/splunk/etc/auth/server.pem

if yes then what should outputs.conf have in sslCertPath = /opt/splunk/etc/auth/?

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 01:32:24 GMT

This should work, as a test, place each part on the required element

outputs.conf --------- [tcpout] defaultGroup = indexers indexAndForward = false [tcpout:indexers] server = x:9997 useSSL = true sslCertPath = $SPLUNK_HOME/etc/auth/server.pem sslPassword = password inputs.conf ----- [splunktcp-ssl:9997] disabled = 0 [SSL] serverCert = $SPLUNK_HOME/etc/auth/server.pem sslPassword = password requireClientCert = false

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 01:44:48 GMT

Thats works!

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 02:46:40 GMT

This indicates that the issue is likely to be certificate or file permissions related.
As a side note, not sure why you have the server cert on the client end.

Can you please confirm the certificate you are trying to configure on the server side (inputs.conf) includes all the certificates in the chain that would be required to fully verify (host all the way back to root ca).

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 02:59:23 GMT

I have found the Splunk documentation quite confusing regarding TLS.

When creating the keys, I followed the same naming conventions used in the Splunk documentation: How to create and sign your own TLS certificates - Splunk Documentation

Which gave me the following files:

myCertAuthCertificate.csr
myCertAuthCertificate.pem
myCertAuthCertificate.srl
myCertAuthPrivateKey.key
myServerCertificate.csr
myServerCertificate.pem
myServerPrivateKey.key

I then did the following command

cat myServerCertificate.pem myServerPrivateKey.key myCertAuthCertificate.pem > mySplkCliCert.pem

In inputs.conf I have the concatenated cert (mySplkCliCert.pem)

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 03:12:00 GMT

If you search index=_internal x509 since the last restart of the indexer, does anything show up

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 03:15:37 GMT

03-16-2023 02:33:18.457 +0000 ERROR X509 [1969765 TcpChannelThread] - X509 certificate (CN=splunkbase.splunk.com,O=Splunk Inc.,L=San Francisco,ST=California,C=US) common name (splunkbase.splunk.com) did not match any allowed names (apps.splunk.com,cdn.apps.splunk.com) host = Splunk-SearchHead-RedHat1source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

03-16-2023 03:01:02.758 +0000 ERROR X509 [2841646 ApplicationUpdateThread] - X509 certificate (CN=splunkbase.splunk.com,O=Splunk Inc.,L=San Francisco,ST=California,C=US) common name (splunkbase.splunk.com) did not match any allowed names (apps.splunk.com,cdn.apps.splunk.com) host = Splunk-Indexer-Redhat1source = /opt/splunk/var/log/splunk/splunkd.logsourcetype = splunkd

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 04:08:19 GMT

Hmm. Looks like there is something wrong with the cert.
Are you creating your own cert due to organisation requirements or for some other reason.
If you are crating "just because" then I might suggest just using the certs and config from earlier that worked.

If it is being done to meet an org requirement, then it looks like the whol

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 04:08:52 GMT

If it is being done to meet an org requirement, then it looks like the whole certificate process may need to be re-visited

Re: Why am I having trouble with TLS?

NJ — Thu, 16 Mar 2023 04:14:31 GMT

Hi @michael_bates_1

Really appreciate your help so far!!!

I need to understand the process of setting up TLS so I can do it for future customers.

Re: Why am I having trouble with TLS?

michael_bates_1 — Thu, 16 Mar 2023 04:28:13 GMT

No problems.

Essentially, until you are really comfortable with it, I always try and stick to the following for customers.
1. Unless you need to use organisational certificates, simply use the ones that Splunk generates.
You can easily regenerate these using a splunk command
bin/splunk createssl
2. Start simple and work your way up - the further down this list, the more error prone and harder to troubleshoot
a) default certs
b) org certs
c) server verification (client verifies the server cert)
d) client verification (server verifies the client cert)
3. Get familar with tools/commands like "openssl s_client" that can be used to verify the certificate exchange, etc.
This can be really helpful to understand where the ssl protocol may be failing
e.g. openssl s_client -connect indexer:9997 -debug -showcerts