Splunk Universal Forwarder TLS certificate update. How to manage in phased manner?

[koshyk]

We have around 3000 UF's talking to Deployment-server and sending data to Indexers using TLS. The current certificate on these clients are going to expire, but the client don't want to update on all 3K servers at the same time 😞

My worry is , say if we upgrade on first 100 clients, the new PEM will be present in the deployment-server/indexers. This means either it will break the 1st 100 or remaining 2900.

1. Is there any clever options which you guys tried out to update certificates in a phased manner?
2. I'm thinking of starting a separate instance of deployment-server to cater for the migrated clients. Any better options would be highly grateful.

[worshamn]

Just use the deployment server to send out a new app to a select group of forwarders that will cause them to connect using a new cert. Though you may want to open up a new listening port on the forwarder or indexer (whichever you are using) so you know that it is working and won't run into conflicts, like 9998. In the app provide an /etc/auth folder with your new certificate and CA, then provide an outputs.conf file in the app something similar to this:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = your_receiving_server:9998
compressed = true
sslCertPath = $SPLUNK_HOME/etc/apps/your_app_name/etc/auth/your_cert_name.pem
sslRootCAPath = $SPLUNK_HOME/etc/apps/your_app_name/etc/auth/your_CA_cert.pem
sslPassword = <your_cert_password>
sslVerifyServerCert = true

I use this for all new UFs coming in, I simply make sure the admins give the UF a deployment.conf file pointed at the deployment server and then have the deployment server hand this app out to all UFs, once it is setup it makes it very easy to change the cert using the deployment server. As I mentioned your receiving forwarder or indexer will need an new listening port, in inputs.conf :

[splunktcp-ssl:9998]
compressed = true
connection_host = ip
rootCA = $SPLUNK_HOME/etc/auth/your_CA_cert.pem
serverCert = $SPLUNK_HOME/etc/auth/your_cert_name.pem
sslPassword = your_cert_password
requireClientCert = false

[koshyk]

hi,
I tried this, but the [SSL] stanza get's impacted for whole of the Server not just the port. So all the existing clients starts sending to the new CA and shows error

[worshamn]

Oops, sorry I should not of had that SSL stanza, those options should be directly under the splunktcp-ssl:9998 stanza, I will edit