For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016
What are the suggested recommendations?
PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.
(Updated: May 19, 2016)
SUMMARY
Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.
Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.
IMPACT
Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.
Expiration of Splunk certificates does not affect:
1) Splunk instances that are in Splunk Cloud
2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).
3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.
4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)
Certificate expiration DOES affect Splunk deployments where:
Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes
RECOMMENDATIONS
There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.
1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.
The script is available at:
http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip
Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version
Please be sure to read the README.txt included in the zip file before running the script.
2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:
Splunk Security: Hardening Standards
3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.
Splunk Security: Hardening Standards
4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.
Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL...
See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate
The files are available here: https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90:
THe FAQ sent by Splunk for expiry on July 21, 2016:
Dear Splunker,
Enclosed are remediation steps for the expiration of default certs shipped
with Splunk 6.2 and earlier.
Please let us know if these steps were helpful.
SUMMARY OF THE ISSUE
In a nutshell, default certificates shipped with 6.2 and earlier versions
of Splunk have expired and will affect communications between Splunk
components:
HTTPS - between browser and Splunk will fail if the following is set:
webconf: EnableSplunkWebSSL, confirm default certs are targeted in
privKeyPath and caCertPath. You will want to swap
out the default certs (https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90) or run the s-renewalcerts scripts.
UF and IDX - This will fail IF SSL is set for the type of connection
between the forwarder and indexers. FYI, dire error messages will be
written to logs even if communications between fwd-idx are working just
fine. You will want to swap out the default certs
(https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90)
or run the s-renewalcerts scripts.
Forwarder (client)
sslVerifyServerCert = false
Indexer (server)
requireClientCert = false
First Response FAQ
Q: Am I using SSL to encrypt between the forwarder and indexer?
A: Run this search on the indexer and a true result will confirm use of
SSL to encrypt communications between the forwarders and indexers indicated.
index=_internal source=metrics.log group=tcpin_connections | dedup
hostname | table hostname sourceIp fwdType version destPort ssl
If ssl=false for all your connections, no further action is necessary.
Q: How do I check the expiration for all certs on the customer's system?
A: A quick way to check all expiration dates for all certs under
$SPLUNK_HOME/etc:
$SPLUNK_HOME/etc/$ find ./ -name '*.pem' -exec openssl x509 -in '{}'
-noout -enddate \;
Q: What are my options for immediate remediation?
A: Run the script provided in the advisory.
The advisory:
https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-lig
ht-and-hunk-pre-63.html
Or, manually swap out the expired default certs downloadable at:
https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90
See steps below.
Q: What if I can't get the script to work? Do we have manual
steps?
A: Yes. Here they are:
1) Stop Splunk
2) Back up (note permissions/ownership, very important!!) and then remove:
· ./etc/auth/ca.pem
· ./etc/auth/cacert.pem
· ./etc/auth/server.pem
3) Copy over the new versions of the following from
https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 to:
· ./etc/auth/ca.pem
· ./etc/auth/cacert.pem
· Do not copy over server.pem
4) Confirm permissions and ownership of 3).
5) Restart Splunk.
A new server.pem is generated. You're done.
Q: How can I tell which SSL parameters have been set in my
configuration files?
A: We recommend using btool along with grep
splunk btool server list --debug | grep blargh
Check for:
[sslConig]
enableSplunkdSSL = true
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
sslVerifyServerCert = true
splunk btool outputs list --debug
[tcpout:splunkssl]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pems
RootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
splunk btool inputs list --debug
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem
Additional Resources
Download the script:
http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip
Splunk Wikis:
http://wiki.splunk.com/Community:SplunkWeb_SSL_DefaultCerts
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts
Thank you,
The Global Customer Support Team
If i run the below command and the ssl is false, do i still need to update the root certificate?
index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl
You should be okay, since that indicates a non-ssl enabled port for that particular connection.
Thanks Weeb!
Same here, I just want to know if I run that search and every line item shows SSL = False, does that mean we do not use the Default Root Certificate?
Could we get an answer to this question? I've run the same thing and get 450 hosts in my env where the ssl column shows false. Does that mean they'll be okay?
If we use the script, it updates ca.pem and cacert.pem but it doesn't update server.pem. If you want to update the server.pem, following are the steps:
1) Generate CSR using default private key available under auth folder:
2) Sign the CSR using cacert.pem
3) Copy content of default private key into newly generated certificate.
4) Copy content of cacert.pem into newly generated certificate.
5) Rename the certificate to server.pem.
6) Restart Splunk.
This will ensure that default server certificate is signed with latest root certificate.
Shameless self promotion of SSL Certificate Checker TA that helps you know when your certificates expire:
I wrote this blog post on how to use the deployment server to push out Splunk's root cert update script. It may be useful to some of you.
No fez required.
http://www.gtri.com/using-splunk-scripted-inputs-beyond-data-collection/
This worked great for me, thanks for posting it!
Thank for letting us know this worked for you!
Hi,
I have a question for a clustered environment and need some clarification. We already have SSL switched on.
For FWD to IDX (or intermediate FWD) connections I will add a new selfsigned CA to the default CA file and will replace the default server.pem with a new server.pem signed by my new CA.
Reboot the FWD or intermediate FWD.
What is the correct order regarding cluster master vs. IDX configuration? Who will create the connection?
I would assume the IDX... in this case I have to setup the IDX first and rebot.
Then the CM and reboot.
Regarding the SH vs. IDX: SH first, then the IDX (which has already been updated...) so SH first, bevor all FWD, IDX, CM. Shut SH down and leave it down until all other instances have been updated...
Summary:
1.) Updating CA file and server.pem on FWD and Reboot FWD (configure outputs.conff with new certs, port 9997)
2.) Updating CA file and server.pem on IDX (configure inputs.conf with new certs, port 9997) and Reboot IDX
3.) Shut SH down and updating new CA file and server.pem (configure server.conf with new certs, port 8089)
4.) Configure IDX (server.conf with new certs, port 8089) and Reboot IDX
5.) Updating CA file and server.pem on ClusterMaster (configure server.conf with new certs, port 8089) and Reboot CM
6.) Startup Search Head
Is this the right order?
the certificates are loaded into splunk at start up time ...not on each connection...
you probably want to shutdown/restart each communicating component ...at the same time
e.g CM<>IDX , SH->IDX/,
i am currently on 6.1 with the default certs. If i upgrade to 6.3 across ALL my indexers, masters, forwarders, search heads etc, will anything else need to be done so that the cert expirations are no longer an issue?
option (4) seems to indicate that i will need to delete $SPLUNK_HOME/etc/auth/ca.pem
and $SPLUNK_HOME/etc/auth/cacert.pem
on each and every pre-6.3 install i have (including all forwarders etc) in order for splunk to generate new certs. can this be done first on my backend (indexers etc.) and then on my forwarders so that i do not have any loss in data ingestion?
with upgrade..the certs are renewed..
"/opt/splunkforwarder/etc/auth/ca.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/cacert.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/server.pem": certificate renewed
$ splunk cmd openssl x509 -in cacert.pem -noout -enddate
notAfter=May 8 19:51:37 2025 GMT
Is there any guidance as to what order each Splunk component (indexer, forwarder, search head, etc) should be updated?
I have deployed lots of UF on my servers. I am just thinking whether I should update the certificate of my Indexer first or my UF(s) first.......
I am afraid that if I update the certificate on my Indexer, then all UF (with the old and going to expire certificate) will disconnect with my Indexer. Thus, no log can be sent to Indexer from UF, until the certificate is updated on these UF(s)......!
I have more than 150 servers deployed with UF....... I don't want to update the certificate 150 times.... please.....
Anyone can give me some suggestions? 😞
Does this also apply to server.pem certificate ? I ran the script provided., passing the argument -serverCert but still there is no change in the server.pem expiry date.
Please advise.
server.pem dates are not renewed.
only the CA (cacert.pem) is renewed
cacert.pem (with its July 21 expire) is embedded in server.pem
to be honest , ive only seen things like external REST calls fall because of this .... but maybe other splunk processes will to.
The script rebuilds the server.pem.... with its original details, but with the new cacertpem(with renewed expiry)
if after running the renewal script. your server.pem expiry date is close approaching... you can rebuild with
splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c hostname
For upgrades from an earlier version to 6.3.x, please remove existing copies of ca.pem and cacert.pem before upgrade.
Steps for Linux:
Hope this helped anyone wondering why their upgrade did not work to change the expiration dates on their default certs.