How to configure SSL certificates to securely forward logs from multiple universal forwarders to our indexer server?

[strive]

Hi,

We have a requirement to forward logs from clients (Splunk universal Forwarders) to a server using SSL (tls1.2)

First Try: We installed same server certificate on both server and clients (as mentioned in the examples in splunk documentation and in splunk blogs). It worked fine.

Change request: Each client should have its own client certificate.

Second Try: We created multiple client certificates, one for each client. Installed those certificates on the client. We started getting a error: connection is not established.

For second try, we followed the below mentioned steps
Step 1: Created a CA Certificate - CACert.pem
Step 2: Created a Server Certificate using the above CA Certificate - ServerCert.pem
Step 3: Created four client certificates using the above CA Certificate - Client1Cert.pem, Client2Cert.pem, Client3Cert.pem, and Client4Cert.pem
Step 4: Installed certificates
Step 5: Restarted Splunk on server and clients.
...... Started seeing connection related errors in splunkd.log ............

Client1 outputs.conf file

[tcpout]
disabled = false
defaultGroup = mac

[tcpout:mac]
server = xxx.xxx.xxx.xxx:YYYY
sslRootCAPath = $SPLUNK_HOME/etc/certs/CACert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/Client1Cert.pem
sslPassword = Client1_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck=commonnametest

Client2 outputs.conf file

[tcpout]
disabled = false
defaultGroup = mac

[tcpout:mac]
server = xxx.xxx.xxx.xxx:YYYY
sslRootCAPath = $SPLUNK_HOME/etc/certs/CACert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/Client2Cert.pem
sslPassword = Client2_privkey_password
sslVerifyServerCert = true
sslCommonNameToCheck=commonnametest

similarly for other clients......

Server inputs.conf file

[SSL]
rootCA = $SPLUNK_HOME/etc/certs/CACert.pem
serverCert = $SPLUNK_HOME/etc/certs/ServerCert.pem
password = server_privkey_password
requireClientCert = true
sslVersions = tls1.2
allowSslRenegotiation = true

[splunktcp-ssl:YYYY]

How will universal forwarder clients validate that the server certificate that is presented is valid? Similarly, how will the server validate that the client certificate that is presented is valid?
What is wrong here? Could you please help.

Thanks,
Strive

[strive · Accepted solution]

requireClientCert = false , makes it work.

As per this link , setting "requireClientCert = true" would require the following conditions to be met :
a) "rootCA" must point to a file containing the CA's public key.
b) The forwarder's server certificate defined by "sslCertPath" in outputs.conf is signed by that CA.
c) The forwarder has the password to read its own certificate ("sslPassword" in outputs.conf).

In our case, we were meeting all the conditions but still we faced issues.

In the same link, there is point which says -- "The purpose of this requireClientCert=true is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer."

So, here is my observation:
requireClientCert = true should be set, when we are using same signed certificate on the server(receiver) and on all the clients (forwarders)