Security

For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

Ellen
Splunk Employee
Splunk Employee

For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016

What are the suggested recommendations?

1 Solution

Ellen
Splunk Employee
Splunk Employee

PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.

(Updated: May 19, 2016)


SUMMARY

Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.

Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.


IMPACT

Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.

Expiration of Splunk certificates does not affect:

1) Splunk instances that are in Splunk Cloud

  • SSL certificates used for Splunk Cloud instances are not the default Splunk certificates
  • Forwarder to Splunk Cloud traffic is not impacted, however, relay forwarders (forwarder to forwarder) can be impacted if you chose to use default Splunk certificates for this communication

2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).

3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.

4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)

Certificate expiration DOES affect Splunk deployments where:

Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes

  • Search Heads
  • Indexers
  • License Masters
  • Cluster Masters
  • Deployers
  • Forwarders

RECOMMENDATIONS

There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.

1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.

The script is available at:

http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version

Please be sure to read the README.txt included in the zip file before running the script.

2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:

Splunk Security: Hardening Standards

3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.

Splunk Security: Hardening Standards

4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.

Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL...

See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate

View solution in original post

weeb
Splunk Employee
Splunk Employee

The files are available here: https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90:

THe FAQ sent by Splunk for expiry on July 21, 2016:

Dear Splunker,

Enclosed are remediation steps for the expiration of default certs shipped
with Splunk 6.2 and earlier.

Please let us know if these steps were helpful.

SUMMARY OF THE ISSUE

In a nutshell, default certificates shipped with 6.2 and earlier versions
of Splunk have expired and will affect communications between Splunk
components:

  1. HTTPS - between browser and Splunk will fail if the following is set:
    webconf: EnableSplunkWebSSL, confirm default certs are targeted in
    privKeyPath and caCertPath. You will want to swap
    out the default certs (https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90) or run the s-renewalcerts scripts.

  2. UF and IDX - This will fail IF SSL is set for the type of connection
    between the forwarder and indexers. FYI, dire error messages will be
    written to logs even if communications between fwd-idx are working just
    fine. You will want to swap out the default certs

(https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90)

or run the s-renewalcerts scripts.

  1. Most Everything else (Deployment Server - Deployment Client, License Master to License Slave, Distributed Search Server-Client, Search Head to Search Peers) will be able to be turned off with the following flags:

Forwarder (client)
sslVerifyServerCert = false

Indexer (server)
requireClientCert = false


First Response FAQ


Q: Am I using SSL to encrypt between the forwarder and indexer?
A: Run this search on the indexer and a true result will confirm use of
SSL to encrypt communications between the forwarders and indexers indicated.

index=_internal source=metrics.log group=tcpin_connections | dedup
hostname | table hostname sourceIp fwdType version destPort ssl

If ssl=false for all your connections, no further action is necessary.


Q: How do I check the expiration for all certs on the customer's system?
A: A quick way to check all expiration dates for all certs under
$SPLUNK_HOME/etc:

$SPLUNK_HOME/etc/$ find ./ -name '*.pem' -exec openssl x509 -in '{}'
-noout -enddate \;


Q: What are my options for immediate remediation?

A: Run the script provided in the advisory.
The advisory:

https://answers.splunk.com/answers/395886/for-splunk-enterprise-splunk-lig
ht-and-hunk-pre-63.html

Or, manually swap out the expired default certs downloadable at:

https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90

See steps below.


Q: What if I can't get the script to work? Do we have manual
steps?
A: Yes. Here they are:
1) Stop Splunk
2) Back up (note permissions/ownership, very important!!) and then remove:

· ./etc/auth/ca.pem

· ./etc/auth/cacert.pem

· ./etc/auth/server.pem
3) Copy over the new versions of the following from

https://splunk.box.com/s/dqylhnq48vmobb46jjcj475mzyu7rv90 to:

· ./etc/auth/ca.pem

· ./etc/auth/cacert.pem

· Do not copy over server.pem
4) Confirm permissions and ownership of 3).
5) Restart Splunk.
A new server.pem is generated. You're done.


Q: How can I tell which SSL parameters have been set in my
configuration files?

A: We recommend using btool along with grep
splunk btool server list --debug | grep blargh
Check for:
[sslConig]
enableSplunkdSSL = true
caCertFile = cacert.pem
caPath = $SPLUNK_HOME/etc/auth
sslVerifyServerCert = true
splunk btool outputs list --debug
[tcpout:splunkssl]
sslCertPath = $SPLUNK_HOME/etc/auth/server.pems
RootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
splunk btool inputs list --debug
[SSL]
rootCA = $SPLUNK_HOME/etc/auth/cacert.pem
serverCert = $SPLUNK_HOME/etc/auth/server.pem

[splunktcp-ssl:]

Additional Resources

Download the script:
http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Splunk Wikis:
http://wiki.splunk.com/Community:SplunkWeb_SSL_DefaultCerts
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

Thank you,

The Global Customer Support Team

newbiesplunk
Path Finder

If i run the below command and the ssl is false, do i still need to update the root certificate?

index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl

weeb
Splunk Employee
Splunk Employee

You should be okay, since that indicates a non-ssl enabled port for that particular connection.

0 Karma

JFrenchsplunk
New Member

Thanks Weeb!

0 Karma

JFrenchsplunk
New Member

Same here, I just want to know if I run that search and every line item shows SSL = False, does that mean we do not use the Default Root Certificate?

0 Karma

japetus
Engager

Could we get an answer to this question? I've run the same thing and get 450 hosts in my env where the ssl column shows false. Does that mean they'll be okay?

hardik_splunk
Splunk Employee
Splunk Employee

If we use the script, it updates ca.pem and cacert.pem but it doesn't update server.pem. If you want to update the server.pem, following are the steps:
1) Generate CSR using default private key available under auth folder:
2) Sign the CSR using cacert.pem
3) Copy content of default private key into newly generated certificate.
4) Copy content of cacert.pem into newly generated certificate.
5) Rename the certificate to server.pem.
6) Restart Splunk.

This will ensure that default server certificate is signed with latest root certificate.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Shameless self promotion of SSL Certificate Checker TA that helps you know when your certificates expire:

https://splunkbase.splunk.com/app/3172/

gtriSplunk
Path Finder

I wrote this blog post on how to use the deployment server to push out Splunk's root cert update script. It may be useful to some of you.
No fez required.
http://www.gtri.com/using-splunk-scripted-inputs-beyond-data-collection/

ckurtz
Path Finder

This worked great for me, thanks for posting it!

0 Karma

gtriSplunk
Path Finder

Thank for letting us know this worked for you!

0 Karma

andre_k2
Engager

Hi,
I have a question for a clustered environment and need some clarification. We already have SSL switched on.

For FWD to IDX (or intermediate FWD) connections I will add a new selfsigned CA to the default CA file and will replace the default server.pem with a new server.pem signed by my new CA.

Reboot the FWD or intermediate FWD.

What is the correct order regarding cluster master vs. IDX configuration? Who will create the connection?
I would assume the IDX... in this case I have to setup the IDX first and rebot.
Then the CM and reboot.

Regarding the SH vs. IDX: SH first, then the IDX (which has already been updated...) so SH first, bevor all FWD, IDX, CM. Shut SH down and leave it down until all other instances have been updated...

Summary:
1.) Updating CA file and server.pem on FWD and Reboot FWD (configure outputs.conff with new certs, port 9997)
2.) Updating CA file and server.pem on IDX (configure inputs.conf with new certs, port 9997) and Reboot IDX
3.) Shut SH down and updating new CA file and server.pem (configure server.conf with new certs, port 8089)
4.) Configure IDX (server.conf with new certs, port 8089) and Reboot IDX
5.) Updating CA file and server.pem on ClusterMaster (configure server.conf with new certs, port 8089) and Reboot CM
6.) Startup Search Head

Is this the right order?

0 Karma

jbarlow_splunk
Splunk Employee
Splunk Employee

the certificates are loaded into splunk at start up time ...not on each connection...
you probably want to shutdown/restart each communicating component ...at the same time
e.g CM<>IDX , SH->IDX/,

0 Karma

ytl
Path Finder

i am currently on 6.1 with the default certs. If i upgrade to 6.3 across ALL my indexers, masters, forwarders, search heads etc, will anything else need to be done so that the cert expirations are no longer an issue?

option (4) seems to indicate that i will need to delete $SPLUNK_HOME/etc/auth/ca.pem and $SPLUNK_HOME/etc/auth/cacert.pem on each and every pre-6.3 install i have (including all forwarders etc) in order for splunk to generate new certs. can this be done first on my backend (indexers etc.) and then on my forwarders so that i do not have any loss in data ingestion?

0 Karma

jbarlow_splunk
Splunk Employee
Splunk Employee

with upgrade..the certs are renewed..

"/opt/splunkforwarder/etc/auth/ca.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/cacert.pem": certificate renewed
"/opt/splunkforwarder/etc/auth/server.pem": certificate renewed

$ splunk cmd openssl x509 -in cacert.pem -noout -enddate
notAfter=May 8 19:51:37 2025 GMT

0 Karma

jackgordon
New Member

Is there any guidance as to what order each Splunk component (indexer, forwarder, search head, etc) should be updated?

0 Karma

charlescywong
New Member

I have deployed lots of UF on my servers. I am just thinking whether I should update the certificate of my Indexer first or my UF(s) first.......

I am afraid that if I update the certificate on my Indexer, then all UF (with the old and going to expire certificate) will disconnect with my Indexer. Thus, no log can be sent to Indexer from UF, until the certificate is updated on these UF(s)......!

I have more than 150 servers deployed with UF....... I don't want to update the certificate 150 times.... please.....

Anyone can give me some suggestions? 😞

0 Karma

splunker12er
Motivator

Does this also apply to server.pem certificate ? I ran the script provided., passing the argument -serverCert but still there is no change in the server.pem expiry date.

Please advise.

jbarlow_splunk
Splunk Employee
Splunk Employee

server.pem dates are not renewed.

only the CA (cacert.pem) is renewed

cacert.pem (with its July 21 expire) is embedded in server.pem
to be honest , ive only seen things like external REST calls fall because of this .... but maybe other splunk processes will to.

The script rebuilds the server.pem.... with its original details, but with the new cacertpem(with renewed expiry)

if after running the renewal script. your server.pem expiry date is close approaching... you can rebuild with

splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c hostname

weeb
Splunk Employee
Splunk Employee

For upgrades from an earlier version to 6.3.x, please remove existing copies of ca.pem and cacert.pem before upgrade.

Steps for Linux:

  1. Stop Splunk
  2. Remove $SPLUNK_HOME/etc/auth/ca.pem
  3. Remove $SPLUNK_HOME/etc/auth/cacert.pem
  4. Upgrade procedure as usual (untar tarball over Splunk home directory)
  5. Start Splunk (this will generate a new ca.pem and cacert.pem files)

Hope this helped anyone wondering why their upgrade did not work to change the expiration dates on their default certs.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!