SSL Forwarding: Why does a Splunk forwarder need its own certificate?

[mlorch]

outputs.conf on forwarder gets its own cert. E.g. something like

[tcpout-server://192.168.1.100:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem
sslPassword = changeme2

For what reason does a forwarder needs its own certificate? Is it for decrypting indexer acknowledgements? Or are there any other reasons?
Because in my understanding, the ca-cert is sufficient for SSL handshake. Right? So if I don't use acknowledgements, I can skip sslCertPath & sslPassword?

This was a very helpful answer for me to understand Splunk's SSL:
How do I set up SSL forwarding ...

Thanks for help!

[cantrell]

The problem has less to due with self-signed certificates and far more to do with weak default algorithms...

[jamesarmitage · Accepted solution]

Take a look at this presentation from .conf 2015:

https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

Page 6 has a nice exchange / function matrix, if you scan the "Forwarding" line you'll see that adding an SSL configuration stanza to a forwarder adds encryption to the data being sent to the indexers, as well as certificate authentication and CN checking.

See pages 7 and 8 for an attack scenario on an unsecured forwarder (provided the REST API is enabled on the forwarder). Pages 18-20 go over the forwarder setup in depth. I believe that indexer acknowledgements are independent of SSL configuration and not related in this case.

The whole presentation is definitely worth reading (and watching, the recording is here: https://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4

[mlorch]

Hi James, hi Duane,
Thanks for your answers and the links to this extremely helpful material! I will work though that.
In my opinion the “throwaway certificate” understanding is really important. So until now we had secured forwarder – indexer communication, but it was some kind of “security by obscurity”. I would appreciate a more detailed documentation by Splunk in the official documentation
Thanks again!

[dwaddle]

Can we be more precise in your definition of "secured". Turning on SSL at all provides some level of "secure" (confidentiality) against casual snooping across the wire - even if you're using self-signed certificates on each end. If this is all you care about, then you've done well.

Next, the question of authentication comes up. Do the forwarders need to confirm the indexer they are talking to is "authentic"? If so, then SSL server authentication via certificate becomes needed. The forwarders need a copy of the CA cert chain signing the indexer's cert so they can verify it.

Continuing on authentication, does the indexer need to confirm the clients sending data to it are "authentic"? If so, the SSL client authentication becomes needed. With a throwaway certificate, it's hard to tell which forwarder is who, the throwaway cert becomes a shibboleth of sorts -- is the forwarder "one of us" or not? If you need to be able to tell forwarder A from forwarder B, then you need a unique certificate for each of them - and you need to be able to figure out how to handle revocation and stuff. (It gets complex)

[gn694]

From watching the recording of your presentation I understand how using the default certificates can lead to authentication issues because of the root CA certificate being the same with every instance of Splunk.

If I do am not concerned with authentication, but want to ensure everything is encrypted, are the default certificates ok to use? Or would using our private CA (with one certificate shared used on all Universal Forwarders) provide some added benefit in terms of data encryption over the default certificates?

[dwaddle]

Thanks James! Appreciate the positive comments on the talk. The main reason a forwarder needs a certificate is because of how the code parsing outputs.conf is written. There is no specific "SSL on or off" flag in outputs.conf - the present of the sslCertPath option in outputs carries the dual semantic of "Turn on SSL" and "Use this client certificate". It's a bit convoluted.

This reason is why in the SSL talk we use a single "throwaway" certificate on all of the forwarders, just to give them the option of turning SSL on without having to deal with managing unique certs and other PKI madness for potentially thousands of systems.

[spl_unker]

Hi @dwaddle  and others

Im stuck with couple of questions and this thread is almost close to answer it.  Im actually working on securing communication between Splunk nodes.

I have 4 forwarders sending data to 3 indexers which is in cluster . I have a 1 deployment server to manage  forwarders . and have 1 cluster master:

Question 1 (securing forwarder->indexers): From the Splunk documentation and this thread i understand we need to have certificates for both forwarders and indexers. But don't understand why it is required on forwarders as well? Having certificates only in indexers does the job right? . Is it because of Splunk configuration or code demands to have it in forwarders as well?

Question 2 (Cluster Master->indexers) : Since we have already in production , is there any impact or precautions that needs to be taken while making communication secured  between CM and indexers. please share any link to document on the configuration . 

Thanks in Advance