For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

[Ellen]

For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016

What are the suggested recommendations?

[weeb]

The files are available here: https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90:

THe FAQ sent by Splunk for expiry on July 21, 2016:

Dear Splunker,

Enclosed are remediation steps for the expiration of default certs shipped
with Splunk 6.2 and earlier.

Please let us know if these steps were helpful.

SUMMARY OF THE ISSUE

In a nutshell, default certificates shipped with 6.2 and earlier versions
of Splunk have expired and will affect communications between Splunk
components:

1. HTTPS - between browser and Splunk will fail if the following is set:
   webconf: EnableSplunkWebSSL, confirm default certs are targeted in
   privKeyPath and caCertPath. You will want to swap
   out the default certs (https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90) or run the s-renewalcerts scripts.

2. UF and IDX - This will fail IF SSL is set for the type of connection
   between the forwarder and indexers. FYI, dire error messages will be
   written to logs even if communications between fwd-idx are working just
   fine. You will want to swap out the default certs

(https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90)

or run the s-renewalcerts scripts.

1. Most Everything else (Deployment Server - Deployment Client, License Master to License Slave, Distributed Search Server-Client, Search Head to Search Peers) will be able to be turned off with the following flags:

Forwarder (client)
sslVerifyServerCert = false

Indexer (server)
requireClientCert = false

---

First Response FAQ

---

Q: Am I using SSL to encrypt between the forwarder and indexer?
A: Run this search on the indexer and a true result will confirm use of
SSL to encrypt communications between the forwarders and indexers indicated.

index=_internal source=metrics.log group=tcpin_connections | dedup
hostname | table hostname sourceIp fwdType version destPort ssl

If ssl=false for all your connections, no further action is necessary.

---

Q: How do I check the expiration for all certs on the customer's system?
A: A quick way to check all expiration dates for all certs under
$SPLUNK_HOME/etc:

$SPLUNK_HOME/etc/$ find ./ -name '*.pem' -exec openssl x509 -in '{}'
-noout -enddate \;

---

Q: What are my options for immediate remediation?

A: Run the script provided in the advisory.
The advisory:

https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-lig
ht-and-hunk-pre-63.html

Or, manually swap out the expired default certs downloadable at:

https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90

See steps below.

---

Q: What if I can't get the script to work? Do we have manual
steps?
A: Yes. Here they are:
1) Stop Splunk
2) Back up (note permissions/ownership, very important!!) and then remove:

· ./etc/auth/ca.pem

· ./etc/auth/cacert.pem

· ./etc/auth/server.pem
3) Copy over the new versions of the following from

https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 to:

· ./etc/auth/ca.pem

· ./etc/auth/cacert.pem

· Do not copy over server.pem
4) Confirm permissions and ownership of 3).
5) Restart Splunk.
A new server.pem is generated. You're done.

---

Q: How can I tell which SSL parameters have been set in my
configuration files?

A: We recommend using btool along with grep
splunk btool server list --debug | grep blargh
Check for:
[sslConig]
enableSplunkdSSL = true
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
sslVerifyServerCert = true
splunk btool outputs list --debug
[tcpout:splunkssl]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pems
RootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
splunk btool inputs list --debug
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

[splunktcp-ssl:]

Additional Resources

Download the script:
http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Splunk Wikis:
http://wiki.splunk.com/Community:SplunkWeb_SSL_DefaultCerts
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

Thank you,

The Global Customer Support Team

[newbiesplunk]

If i run the below command and the ssl is false, do i still need to update the root certificate?

index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl

[weeb]

You should be okay, since that indicates a non-ssl enabled port for that particular connection.

[JFrenchsplunk]

Thanks Weeb!

[JFrenchsplunk]

Same here, I just want to know if I run that search and every line item shows SSL = False, does that mean we do not use the Default Root Certificate?

[japetus]

Could we get an answer to this question? I've run the same thing and get 450 hosts in my env where the ssl column shows false. Does that mean they'll be okay?

[hardik_splunk]

If we use the script, it updates ca.pem and cacert.pem but it doesn't update server.pem. If you want to update the server.pem, following are the steps:
1) Generate CSR using default private key available under auth folder:
2) Sign the CSR using cacert.pem
3) Copy content of default private key into newly generated certificate.
4) Copy content of cacert.pem into newly generated certificate.
5) Rename the certificate to server.pem.
6) Restart Splunk.

This will ensure that default server certificate is signed with latest root certificate.

[jkat54]

Shameless self promotion of SSL Certificate Checker TA that helps you know when your certificates expire:

https://splunkbase.splunk.com/app/3172/

[gtriSplunk]

I wrote this blog post on how to use the deployment server to push out Splunk's root cert update script. It may be useful to some of you.
No fez required.
http://www.gtri.com/using-splunk-scripted-inputs-beyond-data-collection/

[ckurtz]

This worked great for me, thanks for posting it!

[gtriSplunk]

Thank for letting us know this worked for you!

[andre_k2]

Hi,
I have a question for a clustered environment and need some clarification. We already have SSL switched on.

For FWD to IDX (or intermediate FWD) connections I will add a new selfsigned CA to the default CA file and will replace the default server.pem with a new server.pem signed by my new CA.

Reboot the FWD or intermediate FWD.

What is the correct order regarding cluster master vs. IDX configuration? Who will create the connection?
I would assume the IDX... in this case I have to setup the IDX first and rebot.
Then the CM and reboot.

Regarding the SH vs. IDX: SH first, then the IDX (which has already been updated...) so SH first, bevor all FWD, IDX, CM. Shut SH down and leave it down until all other instances have been updated...

Summary:
1.) Updating CA file and server.pem on FWD and Reboot FWD (configure outputs.conff with new certs, port 9997)
2.) Updating CA file and server.pem on IDX (configure inputs.conf with new certs, port 9997) and Reboot IDX
3.) Shut SH down and updating new CA file and server.pem (configure server.conf with new certs, port 8089)
4.) Configure IDX (server.conf with new certs, port 8089) and Reboot IDX
5.) Updating CA file and server.pem on ClusterMaster (configure server.conf with new certs, port 8089) and Reboot CM
6.) Startup Search Head

Is this the right order?

[jbarlow_splunk]

the certificates are loaded into splunk at start up time ...not on each connection...
you probably want to shutdown/restart each communicating component ...at the same time
e.g CM<>IDX , SH->IDX/,

[ytl]

i am currently on 6.1 with the default certs. If i upgrade to 6.3 across ALL my indexers, masters, forwarders, search heads etc, will anything else need to be done so that the cert expirations are no longer an issue?

option (4) seems to indicate that i will need to delete $SPLUNK_HOME/etc/auth/ca.pem and $SPLUNK_HOME/etc/auth/cacert.pem on each and every pre-6.3 install i have (including all forwarders etc) in order for splunk to generate new certs. can this be done first on my backend (indexers etc.) and then on my forwarders so that i do not have any loss in data ingestion?

[jbarlow_splunk]

with upgrade..the certs are renewed..

"/opt/splunkforwarder/etc/auth/ca.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/cacert.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/server.pem": certificate renewed

$ splunk cmd openssl x509 -in cacert.pem -noout -enddate
notAfter=May 8 19:51:37 2025 GMT

[jackgordon]

Is there any guidance as to what order each Splunk component (indexer, forwarder, search head, etc) should be updated?

[charlescywong]

I have deployed lots of UF on my servers. I am just thinking whether I should update the certificate of my Indexer first or my UF(s) first.......

I am afraid that if I update the certificate on my Indexer, then all UF (with the old and going to expire certificate) will disconnect with my Indexer. Thus, no log can be sent to Indexer from UF, until the certificate is updated on these UF(s)......!

I have more than 150 servers deployed with UF....... I don't want to update the certificate 150 times.... please.....

Anyone can give me some suggestions? 😞

[splunker12er]

Does this also apply to server.pem certificate ? I ran the script provided., passing the argument -serverCert but still there is no change in the server.pem expiry date.

Please advise.

[jbarlow_splunk]

server.pem dates are not renewed.

only the CA (cacert.pem) is renewed

cacert.pem (with its July 21 expire) is embedded in server.pem
to be honest , ive only seen things like external REST calls fall because of this .... but maybe other splunk processes will to.

The script rebuilds the server.pem.... with its original details, but with the new cacertpem(with renewed expiry)

if after running the renewal script. your server.pem expiry date is close approaching... you can rebuild with

splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c hostname

[weeb]

For upgrades from an earlier version to 6.3.x, please remove existing copies of ca.pem and cacert.pem before upgrade.

Steps for Linux:

1. Stop Splunk
2. Remove $SPLUNK_HOME/etc/auth/ca.pem
3. Remove $SPLUNK_HOME/etc/auth/cacert.pem
4. Upgrade procedure as usual (untar tarball over Splunk home directory)
5. Start Splunk (this will generate a new ca.pem and cacert.pem files)

Hope this helped anyone wondering why their upgrade did not work to change the expiration dates on their default certs.