Updating expired Server.pem

[mike_k]

We have a distributed Splunk enterprise deployment with the following separate components:

• 1 Search Head
• An Indexer cluster (containing 1 Cluster Master and 2 peer indexers)
• 1 Deployment Server
• 1 License Server

All servers are running on Windows.

Recently we received a message saying "server certificate is now invalid". I am going through the process of updating this certificate.

I have seen some useful comments about just refreshing this as a self-signed cert (Renewing-server-pem-certificate ), however I'm interested in seeing whether i can refresh using certificates signed by our private CA.

Again i have seen some interesting articles on this (such as this: splunk-certificates-master-guide ). However i still have a couple of questions:

1. With my Deployment Server, if i change the server.pem cert on this server, will it break the comms out to my Universal Forwarders? Will i need to update my certificates on my Universal Forwarders immediately or can this be done gradually over time (since i have quite a few UFs)??
2. With my Indexer cluster, if I change the server.pem cert on one of these servers will it break the comms with the rest of the cluster? Is there an recommended order in which to upgrade the certs on Indexer cluster members?
3. Are there any other common Gotcha's when it comes to Indexer Clusters?

Thanks (in anticipation) 🙂

[mike_k]

Thanks for that reply. I have to admit though to still being a tad confused. 🙂

from what i understand server.pem controls all splunkd traffic including traffic between forwarder/indexer as well as traffic indexer/search head. If that is the case ... as soon as i change the certificate on my indexers then communications to other splunk nodes (search heads, cluster master and forwarders) will break until i update their certificates as well?

or is splunk able to still operate with mismatched certificates on the different endpoints? (it certainly seems to still run with the expired certs for the time being)

[aasabatini]

Hi @mike_k 

the search heads to query the indexers do not need to certify, for sending the logs yes, for example the internal logs, this applies to all roles.
for this I recommend to use the server classes on the deployment server to be able to make a more secure update of the certificates.

[aasabatini]

Hi @mike_k 

I hope you are well

before to start your activities on certificates please read this documentation 

https://docs.splunk.com/Documentation/Splunk/8.2.0/Security/ConfigureSplunkforwardingtousesignedcert...

https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

answers for your questions

1. if your change your certificate on your indexer, yes you need to update certifcates on all the uf, you need to manage your forwarders with the server classes, if you have distinct server classes for each role you should avoid to break the communication
2. if you mean with the SH no, the connection between the indexer and the SH is splunk to splunk.
3. like the answers number 1 manage the update with the server classes.

Hope can help

Ale