- Splunk Answers
- :
- Splunk Administration
- :
- Security
- :
- Re: For Splunk Enterprise, Splunk Light, and Hunk ...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
For Splunk Enterprise, Splunk Light and HUNK default root certificates prior to 6.3 will expire on July 21, 2016
What are the suggested recommendations?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
PRODUCT ADVISORY: Pre 6.3, Splunk Enterprise, Splunk Light and HUNK default root certificates expire on July 21, 2016.
(Updated: May 19, 2016)
SUMMARY
Instances of Splunk Enterprise, Splunk Light and HUNK that are older than 6.3 AND that are using the default certificates will no longer be able to communicate with each other after July 21, 2016 unless the certificates are replaced OR Splunk is upgraded to 6.3 or later.
Please note that for all Splunk Enterprise versions, the default root certificate that ships with Splunk is the same root certificate in every download.
That means that anyone who has downloaded Splunk has server certificates that have been signed by the same root certificate and would be able to authenticate to your certificates. To ensure that no one can easily snoop on your traffic or wrongfully send data to your indexers, we strongly recommend that you replace them with certificates signed by a reputable 3rd-party certificate authority.
IMPACT
Failure to replace expired certificates prior to this will result in the immediate cessation of network traffic for any connection which uses them.
Expiration of Splunk certificates does not affect:
1) Splunk instances that are in Splunk Cloud
- SSL certificates used for Splunk Cloud instances are not the default Splunk certificates
- Forwarder to Splunk Cloud traffic is not impacted, however, relay forwarders (forwarder to forwarder) can be impacted if you chose to use default Splunk certificates for this communication
2) Splunk instances that use certificates that are internally generated (self-signed) or obtained from an external Certificate Authority (CA).
3) Splunk instances in your configuration that are upgraded to 6.3 or above and use that version’s root certificates.
4) Splunk instances that do NOT use SSL - (This is the default configuration for forwarder to indexer communication)
Certificate expiration DOES affect Splunk deployments where:
Any or all Splunk instances in your deployment run a release prior to 6.3 and use Splunk default certificates. This includes
- Search Heads
- Indexers
- License Masters
- Cluster Masters
- Deployers
- Forwarders
RECOMMENDATIONS
There are several options that you can take to resolve certificate expiration. You must take action prior to July 21, 2016.
1) Remain at your current Splunk version (pre- 6.3) and manually upgrade the current default root certificates with the provided shell script that is appropriate for your operating system. Note that the shell script only replaces the current default root certificate with a new (cloned) certificate with a future expiration date. The script does not replace a Splunk default certificate with your own certificate.
The script is available at:
http://download.splunk.com/products/certificates/renewcerts-2016-05-05.zip
Update: minor script changes to update messages and remove redirect of stderr to /dev/null when checking OpenSSL version
Please be sure to read the README.txt included in the zip file before running the script.
2) Upgrade all Splunk instances in your environment to 6.3 or above and use self-signed or CA-signed certificate. We strongly recommend this as the most secure option. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure:
Splunk Security: Hardening Standards
3) Remain at your current Splunk version (pre- 6.3) and use self-signed or CA-signed certificate. Replace current default root certificates with your own certificates. Download the following document to learn about hardening your Splunk infrastructure.
Splunk Security: Hardening Standards
4) Upgrade ALL Splunk instances to 6.3 or above and use those default root certificates.
Note: Prior to the upgrade, if in use please remove the existing Splunk default certificate copies of ca.pem and cacert.pem
Refer to: Upgrading my Splunk Enterprise 6.2.x to 6.3.x did not upgrade the expiration dates on my default SSL...
See the following link to learn about adding certificates:
Securing Splunk Enterprise
Use the following procedure to configure default certificates:
Configure Splunk forwarding to use the default certificate
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Actually by default the DS communication is not affected, here is how it works:
outputs.conf handles the config for SSL for communication between the forwarders and indexers, which is not used by default.
server.conf handles the config for splunkd communication, which includes the deployment server. When you look at the server.conf specs, you could easily get confused that this advisory would affect your deployment, since enableSplunkdSSL is set to true by default.
The sslVerifyServerCert parameter is the key, this by default on a Splunk install is false, not forcing SSL communication for splunkd either. So we may surmise that in fact, there is no SSL communication by default forced by Splunk, this is probably by design for just this scenario.
sslVerifyServerCert = true|false
Used by distributed search: when making a search request to another
server in the search cluster.
Used by distributed deployment clients: when polling a deployment
server.
If this is set to true, you should make sure that the server that is
being connected to is a valid one (authenticated). Both the common
name and the alternate name of the server are then checked for a
match if they are specified in this configuration file. A
certificiate is considered verified if either is matched.
Default is false.
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Serverconf
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Currently i'm using 6.3.1V in all the HF, And 6.2.0 in Indexer, SH, DS.
In all the instances "sslVerifyServerCert = false".
Still do I need to update my SSL certificate?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As long as you are not using SSL in your outputs.conf configuration, you should be good to go for communication from forwarders to indexers. As mentioned above, since sslVerifyServerCert=false by default, as long as you haven't changed it on your nodes, you should also be good to go for the rest of your Splunk infrastructure as well.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks mcluver.
And im thinking to upgrade a splunk version and ssl certificate on indexers from 6.2.0 to 6.3.1, Is there any way
1) without any data loss can we upgrade the splunk version on indexers?
2) If we update this SSL certificate will it lead to any data loss?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I forgot whether I enabled SSL on the UF installed on my Windows Servers or not. How do I verify it?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yep
by default, the inter splunk communication should be encrypted and compressed ... but no validation of the cert.
If sslVerifyServerCert = true, there is validation on the cert. Come July 22nd... as the CA cert will have expired.. the verification will fail
there is also this,
http://docs.splunk.com/Documentation/Splunk/latest/Security/AboutsecuringyourSplunkconfigurationwith...
for Inter-Splunk communication , the column "Certificate Authentication" also says not enabled ..
There are a number of Inter-Splunk communication processes that will fail in following setup after July 21 :
*) server .conf has sslVerifyServerCert = true ( the rest of SSLconfig parameters at default settings)
AND
*) certs not renewed via supplied script
examples:
Deployment client > Deployment Server
License Slave> License Master...
Search Heads > Search Peers
.
sample errors
07-22-2016 15:19:28.948 +0100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
07-22-2016 15:20:51.715 +0100 ERROR SSLCommon - Certificate doesn't verify, err=10
07-22-2016 15:20:51.715 +0100 INFO NetUtils - SSL Connection could not be made - server authentication error
07-22-2016 15:20:51.715 +0100 WARN HTTPClient - SSL_ServerAuthError connecting to=nnn.nn.nnn 143:8089
07-22-2016 15:20:51.715 +0100 WARN HTTPClient - Connect to=nnn.nnn.nnn:8089 timed out; exceeded 30sec
07-22-2016 15:20:51.715 +0100 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master: https://nn.nn.nnn.nnn:8089 Connect to=https://nnn.nnn.nn.nnn:8089 timed out; exceeded 30sec’
So in pre 6.3 envs.. to see if deployment could be impacted come july 22..... as well as checking if forwarders/indexers are configured to use the default SSL certs
, need to check server.conf on all servers for , sslVerifyServerCert = true
e.g
splunk btool server list --debug
splunk btool inputs list --debug
splunk btool outputs list --debug
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I second this question. Further, I am curious if it's a problem for Forwarders on a version prior to 6.3 where you are not using SSL communication between the Forwarders and Indexers. Or in other words, does the root cert expiring cause splunkd to not function even while not using SSL features?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
This is what my exact doubt..!!
Can anyone reply for this???
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
How about server certificate(server.pem) ?
Only required action is replacing default root certificate ?
It seems that the script includes the option to replace the server certificate(s-renewcertssh -serverCert) , but what you mentioned above is only root certificate.
Thanks advance.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Yes, I too have the same query.
script execution says :
At least one of -defaultCA, -liveCA, or -serverCert is required;
order should be -defaultCA, -liveCA, -serverCert.
after providing the "-serverCert" as argument ., no change in the server cert it still shows the older expiry date before I ran the script.
Any idea?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Regarding option #2 above, I'm a big fan of http://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPra... / http://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4 which goes through the whole process of removing self-signed certs from end to end.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We've struggled to get end to end ssl working. So many things broke that getting the perfect order for every single instance across the organisation timed exactly right was impossible. As such we had to use the default certificates that came with the product.
We're now in a worse position were we are going to lose a significant volume of data as forwarding hosts will be lost after this date as we don't have contact information for the teams that own those devices (they had project defined requirements to install a forwarder at the time).
There are going to be some very pissed off people because of this.
ICYMI - Check out the latest releases of Splunk Edge Processor
Introducing the 2024 SplunkTrust!
Introducing the 2024 Splunk MVPs!
