Security

For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

Ellen
Splunk Employee
Splunk Employee

For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016

What are the suggested recommendations?

1 Solution

Ellen
Splunk Employee
Splunk Employee

PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.

(Updated: May 19, 2016)


SUMMARY

Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.

Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.


IMPACT

Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.

Expiration of Splunk certificates does not affect:

1) Splunk instances that are in Splunk Cloud

  • SSL certificates used for Splunk Cloud instances are not the default Splunk certificates
  • Forwarder to Splunk Cloud traffic is not impacted, however, relay forwarders (forwarder to forwarder) can be impacted if you chose to use default Splunk certificates for this communication

2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).

3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.

4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)

Certificate expiration DOES affect Splunk deployments where:

Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes

  • Search Heads
  • Indexers
  • License Masters
  • Cluster Masters
  • Deployers
  • Forwarders

RECOMMENDATIONS

There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.

1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.

The script is available at:

http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version

Please be sure to read the README.txt included in the zip file before running the script.

2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:

Splunk Security: Hardening Standards

3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.

Splunk Security: Hardening Standards

4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.

Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL...

See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate

View solution in original post

weeb
Splunk Employee
Splunk Employee

Edit saved! Thanks for catching the error.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

You can also do the following (based on the shell script that Ellen has linked.. Again this is for Splunk < 6.3..

openssl req -new -key ca.pem -x509 -config openssl.cnf -subj '/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com/' -days 3650 > cacert.pem

Then copy the new cacert over the existing Splunk cert, and restart your Splunk Instance.

triest
Communicator

If you get an error about command not found, Splunk bundles openssl

IMHO that best way to use the bundled version is $SPLUNK_HOME/bin/splunk cmd openssl

0 Karma

Ellen
Splunk Employee
Splunk Employee

PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.

(Updated: May 19, 2016)


SUMMARY

Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.

Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.


IMPACT

Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.

Expiration of Splunk certificates does not affect:

1) Splunk instances that are in Splunk Cloud

  • SSL certificates used for Splunk Cloud instances are not the default Splunk certificates
  • Forwarder to Splunk Cloud traffic is not impacted, however, relay forwarders (forwarder to forwarder) can be impacted if you chose to use default Splunk certificates for this communication

2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).

3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.

4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)

Certificate expiration DOES affect Splunk deployments where:

Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes

  • Search Heads
  • Indexers
  • License Masters
  • Cluster Masters
  • Deployers
  • Forwarders

RECOMMENDATIONS

There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.

1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.

The script is available at:

http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version

Please be sure to read the README.txt included in the zip file before running the script.

2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:

Splunk Security: Hardening Standards

3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.

Splunk Security: Hardening Standards

4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.

Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL...

See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate

View solution in original post

karthikklv
Engager

HI All,

We are using Splunk Enterprise 6.3.2. But, we still got an email saying: Product Advisory: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016

As we are above version 6.3, could any one of you tell me, why we still got this email.
Are we missing to consider anything here?

0 Karma

karthikklv
Engager

We can check the end date for root certificates. Run the command on Splunk Indexers under

$SPLUNK_HOME/etc/auth

openssl x509 -enddate -noout -in ca.pem
openssl x509 -enddate -noout -in cacert.pem

0 Karma

keithyap
Path Finder

I tried using the script but seem to be having an issue with the server.pem cert.
All the other certs have their validity dates extended to 2026, but ther server.pem shows that it will expire in 2017.

Is this the default behavior for this? My Splunk instance is on a Windows Server.

wittenst123
Engager

I'm having a similar issue with the server.pem cert (where the script sets it up to expire in three years instead of ten). We're on Red Hat Linux

0 Karma

davidpaper
Contributor

Ellen, can you update the answer to include specific verbiage around the communication between the forwarders and the Deployment Server? Based on the comments by joshd below, if the customer has not turned on client certificate validation, then the communication between forwarder to DS will still work after 7/21/16.

0 Karma

dhavamanis
Builder

Currently we are using self-signed certificate for data transfer from forwarder to Splunk instances. Do we need to regenerate certificate again after upgrading to Splunk 6.4 and use the new certificates in forwarder?. We followed the below post for generating new self-signed certificates,

https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certific...

We have enabled https option for indexer and master node to access through UI, that certificate expiration date says,

Issues By
Common Name(CN) SplunkCommonCA
Organization(O) Splunk

Period of Validity

Begins On 10/22/2014
Expires On 10/21/2017

Can you please share your thoughts?.

0 Karma

czheng
New Member

In Recommendation 1), will running the script require a restart of the server or Splunk?

0 Karma

jbarlow_splunk
Splunk Employee
Splunk Employee

yes. latest version of the script/readme now states this

0 Karma

triest
Communicator

If you have bash installed, its easy to check certificate expiration dates:

ports="8089 8443 8191 8065 5100 5555 9997"; SPLUNK_HOME="/opt/splunk"; openssl="${SPLUNK_HOME}/bin/openssl"; timeout=15 ; for port in ${ports} ; do expiration=$(echo | timeout ${timeout} ${openssl} s_client -connect 127.0.0.1:${port} 2>/dev/null | ${openssl} x509 -noout -dates 2>/dev/null | grep notAfter); echo ${port} ${expiration:-non-SSL, timeout, or error} ; done

There may be a couple of things you need to modify
ports="8089 8443 8191 8065 5100 5555 9997";

ports needs to be a space separated list of ports to check; I pulled this from a combination of our search heads and indexers.

SPLUNK_HOME="/opt/splunk";
This one should be completely foreign to any Splunk administrator 🙂 but update the path if need be

timeout=15
I use the timeout command so openssl won't hang if a port isn't open or doesn't use SSL. This gives the openssl's s_client 15 seconds to return the cert. That was plenty of time on our systems, but you may need to increase it.

It shouldn't be hard to write a very similar thing for Windows since Splunk bundles openssl; I just don't have a windows box handy to use to attempt to write one. If there's a request and no one else volunteers I'll spin up a VM tonight.

triest
Communicator

Is using a self-signed certificate enough?

Under recommendations, option 2 states: "Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate."

Why the need to upgrade Splunk to 6.3 or above for option 2? Why isn't the second part sufficient? Especially since under impact it explicitly states "[...] does NOT affect Splunk deployments where: 1) Your configuration uses certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA)"

My assumption is using self-signed certificates would be enough; my concern is there's a place where we aren't replacing the certificate (e.g. mongo or on 8065).

dwaddle
SplunkTrust
SplunkTrust

Good question. I would say it's an oversight. If you've converted over to where you are using your own internal CA, or a public CA - then the expiration of Splunk's certs truly doesn't matter at all, regardless of your Splunk version. Now, when your own certificates expire you'll be in a pickle. But you should know how to manage that already.

0 Karma

dsafian
Engager

Will there be a problem with forwarders running pre-6.3 sending data when the indexers and other infrastructure components are already at 6.3 or later?

jmaher_splunk
Splunk Employee
Splunk Employee

Pre-6.3 forwarders that are configured to use the default certificates will be impacted. If you are not using SSL, you will not be impacted by expiring root certs.

0 Karma

mcluver
Path Finder

From my understanding, if you're utilizing a deployment server with forwarders then you will also be impacted. Correct?

0 Karma

weeb
Splunk Employee
Splunk Employee

mcluver: And if you are using SSL to encrypt between the DS server and the DS client.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

When the automatically generated root certificate -- and all of its child certificates -- expires, all things using SSL will be broken. By DEFAULT, this does NOT include the forwarding of data but DOES include DS client -> DS server.

Good timing I suppose for the Splunk Trust virtual.conf Webex on SSL best practices! We cover ALL of this, in detail. April 28, 2016 at 9AM PT / 12PM ET / 16:00 GMT. http://www.meetup.com/Splunk-Meetups/events/230551134/

ericlarsen
Path Finder

How would you enable SSL between Forwarders and the Deployment Server? The deploymentclient.conf spec doesn't contain SSL attributes.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!