topic Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations? in Security

For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

Ellen — Mon, 18 Apr 2016 23:56:01 GMT

For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016

What are the suggested recommendations?

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

Ellen — Tue, 19 Apr 2016 00:06:07 GMT

PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.

(Updated: May 19, 2016)

SUMMARY

Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.

Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.

IMPACT

Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.

Expiration of Splunk certificates does not affect:

1) Splunk instances that are in Splunk Cloud

SSL certificates used for Splunk Cloud instances are not the default Splunk certificates
Forwarder to Splunk Cloud traffic is not impacted, however, relay forwarders (forwarder to forwarder) can be impacted if you chose to use default Splunk certificates for this communication

2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).

3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.

4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)

Certificate expiration DOES affect Splunk deployments where:

Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes

Search Heads
Indexers
License Masters
Cluster Masters
Deployers
Forwarders

RECOMMENDATIONS

There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.

1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.

The script is available at:

http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip

Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version

Please be sure to read the README.txt included in the zip file before running the script.

2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:

Splunk Security: Hardening Standards

3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.

Splunk Security: Hardening Standards

4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.

Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL certs, why?

See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

dwaddle — Tue, 19 Apr 2016 02:29:56 GMT

Regarding option #2 above, I'm a big fan of http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPractices.pdf / http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4 which goes through the whole process of removing self-signed certs from end to end.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

esix_splunk — Tue, 19 Apr 2016 03:55:01 GMT

You can also do the following (based on the shell script that Ellen has linked.. Again this is for Splunk < 6.3..

openssl req -new -key ca.pem -x509 -config openssl.cnf -subj '/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com/' -days 3650 > cacert.pem

Then copy the new cacert over the existing Splunk cert, and restart your Splunk Instance.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

jrlee — Tue, 26 Apr 2016 11:33:57 GMT

How about server certificate(server.pem) ?
Only required action is replacing default root certificate ?
It seems that the script includes the option to replace the server certificate(s-renewcertssh -serverCert) , but what you mentioned above is only root certificate.

Thanks advance.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

weeb — Tue, 26 Apr 2016 20:05:20 GMT

The following steps will update the expiration date to 10 years into the future for existing key and append it to the existing cacert.pem certificate.

Stop Splunk
Run the following:

$ openssl req -new -key ca.pem -x509 -days 3650 > cacert.pem
You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:US State or Province Name (full name) []:CA Locality Name (eg, city) [Default City]:San Francisco Organization Name (eg, company) [Default Company Ltd]:Splunk Organizational Unit Name (eg, section) []:SplunkCommonCA Common Name (eg, your name or your server's hostname) []:SplunkCommonCA Email Address []:support@splunk.com
$ cat cacert.pem > ca.pem
Start Splunk.
Confirm by running the following:

$ openssl x509 -in "ca.pem" -text -noout
$ openssl x509 -in "cacert.pem" -text -noout

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

dsafian — Wed, 27 Apr 2016 14:18:58 GMT

Will there be a problem with forwarders running pre-6.3 sending data when the indexers and other infrastructure components are already at 6.3 or later?

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

mcluver — Wed, 27 Apr 2016 17:52:16 GMT

I second this question. Further, I am curious if it's a problem for Forwarders on a version prior to 6.3 where you are not using SSL communication between the Forwarders and Indexers. Or in other words, does the root cert expiring cause splunkd to not function even while not using SSL features?

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

jmaher_splunk — Wed, 27 Apr 2016 20:03:53 GMT

Pre-6.3 forwarders that are configured to use the default certificates will be impacted. If you are not using SSL, you will not be impacted by expiring root certs.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

mcluver — Wed, 27 Apr 2016 21:09:44 GMT

From my understanding, if you're utilizing a deployment server with forwarders then you will also be impacted. Correct?

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

weeb — Wed, 27 Apr 2016 21:45:50 GMT

mcluver: And if you are using SSL to encrypt between the DS server and the DS client.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

triest — Thu, 28 Apr 2016 03:09:02 GMT

Is using a self-signed certificate enough?

Under recommendations, option 2 states: "Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate."

Why the need to upgrade Splunk to 6.3 or above for option 2? Why isn't the second part sufficient? Especially since under impact it explicitly states "[...] does NOT affect Splunk deployments where: 1) Your configuration uses certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA)"

My assumption is using self-signed certificates would be enough; my concern is there's a place where we aren't replacing the certificate (e.g. mongo or on 8065).

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

dwaddle — Thu, 28 Apr 2016 05:20:00 GMT

Good question. I would say it's an oversight. If you've converted over to where you are using your own internal CA, or a public CA - then the expiration of Splunk's certs truly doesn't matter at all, regardless of your Splunk version. Now, when your own certificates expire you'll be in a pickle. But you should know how to manage that already.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

dwaddle — Thu, 28 Apr 2016 05:23:36 GMT

When the automatically generated root certificate -- and all of its child certificates -- expires, all things using SSL will be broken. By DEFAULT, this does NOT include the forwarding of data but DOES include DS client -> DS server.

Good timing I suppose for the Splunk Trust virtual.conf Webex on SSL best practices! We cover ALL of this, in detail. April 28, 2016 at 9AM PT / 12PM ET / 16:00 GMT. http://www.meetup.com/Splunk-Meetups/events/230551134/

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

triest — Thu, 28 Apr 2016 15:44:29 GMT

Is $SPLUNK_HOME/etc/auth/openssl really correct?

Just FYI that doesn't exist on our systems; I would use /opt/splunk/bin/splunk cmd openssl

The executable is typically /opt/splunk/bin/openssl ; on some systems that works, but others I have to use /opt/splunk/bin/splunk cmd openssl due to needing to have the correct path to load libraries.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

triest — Thu, 28 Apr 2016 15:49:04 GMT

If you get an error about command not found, Splunk bundles openssl

IMHO that best way to use the bundled version is $SPLUNK_HOME/bin/splunk cmd openssl

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

triest — Tue, 29 Sep 2020 09:34:24 GMT

If you have bash installed, its easy to check certificate expiration dates:

ports="8089 8443 8191 8065 5100 5555 9997"; SPLUNK_HOME="/opt/splunk"; openssl="${SPLUNK_HOME}/bin/openssl"; timeout=15 ; for port in ${ports} ; do expiration=$(echo | timeout ${timeout} ${openssl} s_client -connect 127.0.0.1:${port} 2>/dev/null | ${openssl} x509 -noout -dates 2>/dev/null | grep notAfter); echo ${port} ${expiration:-non-SSL, timeout, or error} ; done

There may be a couple of things you need to modify
ports="8089 8443 8191 8065 5100 5555 9997";

ports needs to be a space separated list of ports to check; I pulled this from a combination of our search heads and indexers.

SPLUNK_HOME="/opt/splunk";
This one should be completely foreign to any Splunk administrator 🙂 but update the path if need be

timeout=15
I use the timeout command so openssl won't hang if a port isn't open or doesn't use SSL. This gives the openssl's s_client 15 seconds to return the cert. That was plenty of time on our systems, but you may need to increase it.

It shouldn't be hard to write a very similar thing for Windows since Splunk bundles openssl; I just don't have a windows box handy to use to attempt to write one. If there's a request and no one else volunteers I'll spin up a VM tonight.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

JScordo — Thu, 28 Apr 2016 15:59:37 GMT

Will this have any impact on Splunk Cloud customers?

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

yannK — Thu, 28 Apr 2016 18:40:50 GMT

Remark specific for SplunkCloud customers.
The SSL certificates used for the outputs.conf are not the stock splunk certificate, therefore will not expire. And the forwarding to splunkcloud will continue.

For non cloud customer, the forwarding will be impacted by the expiration of the default cert if you are using splunktcp-ssl.

[EDIT after keen remarks and testing from joshd]

The splunkd and splunkweb processes will still work using the expired certificate on the forwarders.
The only situation to be aware is if you have setup your servers to validate the certificates (by example a deployment-server, or an API connector). If this is the case, we can assume that this is not the default behavior, and that you already switched the certificates to your own, and are already managing them.

Be healthy upgrade your forwarders.

Re: For Splunk Enterprise, Splunk Light, and Hunk pre 6.3, default root certificates expire on July 21, 2016 - Recommendations?

joshd — Tue, 29 Sep 2020 09:35:00 GMT

Are you sure about the side issues with the Universal Forwarders? What level of testing has been performed?

Right now in my lab we have the following:

Universal Forwarder v6.2.9 (ip-172-31-17-125)
Deployment Server v6.2.9 (ip-172.31.17.127)
Search Head v6.3.2 (ip-172.31.17.128)
Indexer v6.3.2 (ip-172.31.17.124)

The system time on all instances have been set to beyond the expiry date of the certificate (July 21).

[root@ip-172-31-17-125 auth]# openssl x509 -enddate -noout -in cacert.pem
notAfter=Jul 21 17:12:19 2016 GMT
[root@ip-172-31-17-125 auth]# date
Fri Jul 22 22:46:21 UTC 2016

As expected the Splunk SSL forwarding fails connectivity to the Indexer with the Indexer returning a message to check the certificate expiry:

07-22-2016 22:59:00.941 +0000 ERROR TcpOutputFd - Connection to host=172.31.17.124:9997 failed. sock_error = 0. SSL Error = error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

But I am not observing any issues with DS communication. It actually completes the handshake succesfully with the DS:

[splunk@ip-172-31-17-125 ~]$ splunk display deploy-client
Deployment Client is enabled.
[splunk@ip-172-31-17-125 ~]$ splunk show deploy-poll
Deployment Server URI is set to "172.31.17.127:8089".

07-22-2016 22:58:12.841 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
07-22-2016 22:58:17.846 +0000 INFO HttpPubSubConnection - SSL connection with id: connection_172.31.17.125_8089_ip-172-31-17-125.us-west-2.compute.internal_ip-172-31-17-125_uf-6.2
07-22-2016 22:58:17.850 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.31.17.125_8089_ip-172-31-17-125.us-west-2.compute.internal_ip-172-31-17-125_uf-6.2
07-22-2016 22:58:24.842 +0000 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_172.31.17.125_8089_ip-172-31-17-125.us-west-2.compute.internal_ip-172-31-17-125_uf-6.2
07-22-2016 22:58:24.843 +0000 INFO DC:HandshakeReplyHandler - Handshake done.

On the DS side I can see the client phone in, I can assign it to serverclasses and successfully have it download and install deployment applications, even performing a restart after install.

From the CLI/API perspective, I can run commands on the Universal Forwarder to interact as a normal admin might without issue:

[splunk@ip-172-31-17-125 ~]$ splunk disable deploy-client
Your session is invalid. Please login.
Splunk username: admin
Password:
Deployment Client is disabled.
You need to restart the Splunk Server (splunkd) for your changes to take effect.
[splunk@ip-172-31-17-125 ~]$ splunk edit user admin -password doesthiswork
Your session is invalid. Please login.
Splunk username: admin
Password:
User admin edited.

From the Search Head I can actually add the DS as a search peer, further showing splunk-to-splunk API access across 8089 is successful:

[splunk@ip-172-31-17-128 ~]$ splunk list search-server
Server at URI "172.31.17.124:8089" with status as "Up"
Server at URI "172.31.17.127:8089" with status as "Up"

UPDATE 1: Forgot to add, that we also in the lab spun up a DS with 6.3.2 and observe the same success with communication between the UF on 6.2.9 and DS on 6.3.2.