Changing root CA / server certificates on cluster

[splunkreal]

Hello,
we will need to change root CA/server certificates on all our splunk cluster, forwarders will also be affected.
Also we use TCP-SSL on our HF with various network appliances, I guess they will be affected?
Is there good practice or guideline that you could provide?
Thanks for your help.

[kknairr]

@splunkreal Yes, changing the root CA and server certificates in a Splunk deployment affects all cluster members, forwarders, and any external appliances using TCP‑SSL. The key is to plan the migration carefully, so you don’t break connectivity. Highly recommend you connect with Splunk support if it's a critical environment using SSL/Secure connections widely across the cluster and with external appliances.

General Approach could be: 

• Add the new root CA to the trust store first by placing the new certificate on the auth folder (on all Splunk nodes and appliances). This ensures both old and new certs are trusted during the transition.

  • File path: $SPLUNK_HOME/etc/auth/ 

  • Edit the [sslConfig] stanza to reference both the old and new CA files. Splunk supports a comma‑separated list of CA certs, so you can trust multiple roots during migration.

  • Update outputs.conf (for UF/HF) or inputs.conf (for HF receiving TCP‑SSL) to reference the new CA file.

• Issue new server certificates signed by the new CA, with proper CN/FQDN matching.
• Roll out certificates gradually and you may follow the below order.
• Cluster Manager → Indexers → Search Heads → Forwarders → Heavy Forwarders → External appliances.
• Test connectivity with the new CA cert using below openssl commands. This confirms the new CA is trusted

openssl s_client -connect <splunk_host>:9997 -CAfile newRootCA.pem

• Remove the old root CA only after all nodes and clients have switched.

References:

How to Configure Splunk Forwarding to Use Your Own Certificate | Splunk

Configure Splunk Web to use TLS certificates | Splunk Enterprise (last updated 2025-07-04T03:47:01.9...

[PickleRick]

Well, no. At least not entirely. CA certificate affects only a specific connection where it is used. You _can_ (although it's rarely done this way) use several different CA certificates for different purposes (for example, another certificate to present on your inputs and another to use to authenticate on your outputs).

But yes, I do agree that it requires careful planning.

Luckily, Splunk allows for multiple CA certificates so with a bit of care migration to another CA can be quite "easy" (as if anything around TLS was ever easy).

@splunkreal Yes, I mean sslRootCAPath in the part of _adding_ new CA (not replacing the old one because then you immediately stop trusting all old certificates).

But if you don't verify the certs at all, why bother in the first place?

[splunkreal]

Hello @kknairr thanks for detailed explanation, especially sslRootCAPath, no need to put all rootCAs in single file but in the directory then?

[PickleRick]

As with any decent TLS implementation - the way to go about migrating from one root CA cert to another is to add the new root CA cert to the trusted certs store, migrating the relevant subject certificates (or just letting them naturally expire and just issue new ones with the new CA) and then optionally remove the old CA.

The details will obviously depend on the detailed setup on where (and on which end) you use certs. 

The general idea is in the first paragraph. For detailed plan, if you don't feel confident enough to do it on your own, I'd suggest to reach out to your local Splunk partner to thoroughly review your infrastructure and go through the process with you.

[splunkreal]

Hello @PickleRick thanks, do you mean using sslRootCAPath?

By default ssl verify is disabled so is it necessary?

Thanks.