How to add new forwarders when requireClientCert is true?

[R15]

Hi, 

We recently set up (mostly) mTLS, just have a generic forwarder cert. But with the cert being deployed via an app on the deployment server, which has requireClientCert=true, new forwarders can't connect to grab the apps. I disabled it temporarily and everything is fine now, but how is this usually handled long term?

[SK99]

Are you using "sslCommonNameToCheck"; this name should be defined in your cert installed on DS for validating the connections from Forwarders. 

below for your reference...

sslCommonNameToCheck = <comma-separated list>
* One or more X.509 standard Common Names of the server certificate which splunkd,
as a client, checks against when it connects to a server using TLS.
* The Common Name (CN) is an X.509 standard field in a certificate that identifies the
host name that is associated with the certificate.
* The CN can be a short host name or a fully qualified domain name. For example,
the CN can be one of "example", "www.example.com", or "example.com".
* If the client cannot match the CN in the certificate that the server presents,
then the client cannot authenticate the server, and terminates the session
negotiation immediately.
* For this setting to have any affect, the 'sslVerifyServerCert' setting must have
a value of "true".
* This setting is optional.
* No default (no common name checking).

[livehybrid]

Hi @R15 

This is a little bit of a chicken and egg scenario, your clients cant connect to the DS to get the app containing the cert because they require the cert to connect to the DS! 

The way I would probably tackle this is to have a cert which is deployed as part of your forwarder installation script that will allow it to connect and then pull down an app with higher precedence which would then be used moving forwards. 

The other way would be what you're already doing, you could temporarily disable the mTLS which would allow the fowarder to obtain the cert.

Ultimately it depends on your environment and what youre most comfortable with. 

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[R15]

Would you use the same cert? I'm assuming this would need to be in an app so another app could take precedence. I don't think we have deployment scripts currently, I believe we're just cloning a golden image that has splunk installed but I'll ask. 

I thought about scripting disabling/enabling requireClientCert on the DS regularly, but some hosts are not always online and may still be missed. It seems silly to have to do that constantly. 

[PickleRick]

Using the same cert across whole environment doesn't make mufh sense. The cert is supposed to be tied to a specific subject. If you want to have a static pre-shared secret, you can just use pass4SymmKey in DS configuration and set the same one on UFs.

But you also need to preconfigure your UFs with it because without it they won't be able to access the DS.

[R15]

I know, I was lucky to get individual certs for each splunk box. We're going with one cert for all forwarders.