outputs.conf on forwarder gets its own cert. E.g. something like
[tcpout-server://192.168.1.100:9997] sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem sslPassword = changeme2
For what reason does a forwarder needs its own certificate? Is it for decrypting indexer acknowledgements? Or are there any other reasons?
Because in my understanding, the ca-cert is sufficient for SSL handshake. Right? So if I don't use acknowledgements, I can skip sslCertPath & sslPassword?
This was a very helpful answer for me to understand Splunk's SSL:
How do I set up SSL forwarding ...
Thanks for help!
Take a look at this presentation from .conf 2015:
Page 6 has a nice exchange / function matrix, if you scan the "Forwarding" line you'll see that adding an SSL configuration stanza to a forwarder adds encryption to the data being sent to the indexers, as well as certificate authentication and CN checking.
See pages 7 and 8 for an attack scenario on an unsecured forwarder (provided the REST API is enabled on the forwarder). Pages 18-20 go over the forwarder setup in depth. I believe that indexer acknowledgements are independent of SSL configuration and not related in this case.
The whole presentation is definitely worth reading (and watching, the recording is here: https://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4
Thanks James! Appreciate the positive comments on the talk. The main reason a forwarder needs a certificate is because of how the code parsing outputs.conf is written. There is no specific "SSL on or off" flag in outputs.conf - the present of the
sslCertPath option in outputs carries the dual semantic of "Turn on SSL" and "Use this client certificate". It's a bit convoluted.
This reason is why in the SSL talk we use a single "throwaway" certificate on all of the forwarders, just to give them the option of turning SSL on without having to deal with managing unique certs and other PKI madness for potentially thousands of systems.
Hi James, hi Duane,
Thanks for your answers and the links to this extremely helpful material! I will work though that.
In my opinion the “throwaway certificate” understanding is really important. So until now we had secured forwarder – indexer communication, but it was some kind of “security by obscurity”. I would appreciate a more detailed documentation by Splunk in the official documentation
Can we be more precise in your definition of "secured". Turning on SSL at all provides some level of "secure" (confidentiality) against casual snooping across the wire - even if you're using self-signed certificates on each end. If this is all you care about, then you've done well.
Next, the question of authentication comes up. Do the forwarders need to confirm the indexer they are talking to is "authentic"? If so, then SSL server authentication via certificate becomes needed. The forwarders need a copy of the CA cert chain signing the indexer's cert so they can verify it.
Continuing on authentication, does the indexer need to confirm the clients sending data to it are "authentic"? If so, the SSL client authentication becomes needed. With a throwaway certificate, it's hard to tell which forwarder is who, the throwaway cert becomes a shibboleth of sorts -- is the forwarder "one of us" or not? If you need to be able to tell forwarder A from forwarder B, then you need a unique certificate for each of them - and you need to be able to figure out how to handle revocation and stuff. (It gets complex)
From watching the recording of your presentation I understand how using the default certificates can lead to authentication issues because of the root CA certificate being the same with every instance of Splunk.
If I do am not concerned with authentication, but want to ensure everything is encrypted, are the default certificates ok to use? Or would using our private CA (with one certificate shared used on all Universal Forwarders) provide some added benefit in terms of data encryption over the default certificates?
The problem has less to due with self-signed certificates and far more to do with weak default algorithms...