UF -> HF SSL Configuration Weirdness

[elaborateGecko]

Hello, 

I'm currently working on configuring SSL from a UF sitting on a Windows server to a HF running on RHEL 7. I am using third party certs that I obtained from my lab windows PKI environment that has two tiers of CA (RootCA/SubCA) with the HF having a signed cert (.pem file) as well as each UF sharing a single cert that was signed by the same Subordinated CA.

I've managed to successfully achieve/confirm this SSL connection is happening using the Validate your configuration doc by Splunk, but the configuration seems to contradict itself so I would appreciate some insight into where I may have gone wrong that resulted in this. 

The contradiction is within the requireClientCert parameter on the Splunk HF, as well as the clientCert parameter on the Universal Forwarder. I have tested several times after restarting the Splunk service on both workstations and this connection ONLY works when I have both clientCert configured within the outputs.conf as well as requireClientCert = false. If either of these variables are changed or I remove the clientCert parameter (even though it should be technically not required) I have a connection error. 

For example, with the configuration below, if I changed requireClientCert = true the entire connection would fail, even though I believe the clientCert .pem file is configured correctly. 

My certificate chains for each host are as follows:
HF: 
adcs.pem (shared with clients via deployment server, to be used for sslRootCaPath parameter)

<intermediate (subordinate/issuing) ca certificate>
<root ca certificate>

heavyforwarder01.pem (certificate chain to be used for serverCert parameter)

<hf01.pem cert issued by subordinate ca> 
<decrypted rsa key> 
<subca_pub.pem> 
<rootca_pub.pem> 

UF: 
server.conf

sslRootCAPath = C:/path/to/adcs.pem

universalforwarder01.pem (certificate chain to be used for clientCert parameter)

<uf01.pem certificate issued by subordinate CA> 
<encrypted rsa key for cert> 
<subca_pub.pem> 
<rootca_pub.pem> 

Configuration for outputs.conf on Universal Forwarder:

[tcpout]

defaultGroup=splhf01

[tcpout:splhf01]

disabled=0

server = splhf01.domain.local:9998

clientCert = C:\path\to\universalforwarder01.pem

sslPassword = <redacted>

useClientSSLCompression = true

sslCommonNameToCheck = splhf01.domain.local

sslVerifyServerCert = true

Configuration for server.conf on Universal forwarder:

[sslConfig]

sslRootCAPath = C:\path\to\combined\adcs.pem

Configuration for inputs.conf on Heavy Forwarder/Indexer:

[default]

host = splhf01

[splunktcp:9997]

disabled = 0

[splunktcp-ssl:9998]

disabled = 0

[SSL]

serverCert = /opt/splunk/etc/auth/ssl/s2s/heavyforwarder01.pem

requireClientCert = false

sslVersions *,-ssl2

sslCommonNameToCheck = splhf01.domain.local

Configuration for Heavy forwarder server.conf

<..>

[sslConfig]

sslRootCAPath = /path/to/combined/adcs.pem

<...>

[PickleRick]

The TLS configuration can be painful at times because it's relatively sensitive to small mistakes and if you make those mistakes the config gets "kinda working" but does something else that you'd expect it to.

For example, Splunk tends to fall back to its default certs if it can't load the one supplied by you.

So check your errors for both sides of the connection.

Try setting up a HF first and verify the connectivity with simple openssl s_client supplying your certificate.

If you confirm that HF is really listening with TLS on a given port and is indeed authenticating client by cert move on to configure UF as the client. Don't do both at once because then you don't know which side you should be troubleshooting.

[elaborateGecko]

Thank you for your response, you've built up quite the reputation and your answers have helped me countless times before so thank you for that as well.  

While I thought I had combed both sides of the configuration (hf/uf), when I went back this time I did find a certificate error (unsupported certificate purpose), so I'm very glad you pointed me there again. 

I will investigate a bit further and post the solution to this thread if/when I find it

[PickleRick · Accepted solution]

That's also one of the typical issues with third party CA-s. Unless you're careful and insist on certs for client authentication they often issue server authentication only certs (sometimes even if you provide both usages in CSR) - it's still not that common to use mutual-TLS so people are not used to client certs.

[elaborateGecko]

PickleRick is certainly correct. I have since remediated the issue and it mainly had to do with the purpose that each of the aforementioned certificates were used for. 

Client certs that will be on the UF need to be multi-purpose, with x509 extended key usage as well as extensions for TLS web server authentication as well as TLS web client authentication. 

Unfortunately I have seen scant documentation that highlights this requirement, but this is forgivable now that I'm aware it's not exactly a common practice for this to be enabled at most organizations, even larger enterprises.