Getting Data In

SSL Forwarding: Why does a Splunk forwarder need its own certificate?

mlorch
Path Finder

outputs.conf on forwarder gets its own cert. E.g. something like

[tcpout-server://192.168.1.100:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem
sslPassword = changeme2

For what reason does a forwarder needs its own certificate? Is it for decrypting indexer acknowledgements? Or are there any other reasons?
Because in my understanding, the ca-cert is sufficient for SSL handshake. Right? So if I don't use acknowledgements, I can skip sslCertPath & sslPassword?

This was a very helpful answer for me to understand Splunk's SSL:
How do I set up SSL forwarding ...

Thanks for help!

1 Solution

jamesarmitage
Path Finder

Take a look at this presentation from .conf 2015:

https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

Page 6 has a nice exchange / function matrix, if you scan the "Forwarding" line you'll see that adding an SSL configuration stanza to a forwarder adds encryption to the data being sent to the indexers, as well as certificate authentication and CN checking.

See pages 7 and 8 for an attack scenario on an unsecured forwarder (provided the REST API is enabled on the forwarder). Pages 18-20 go over the forwarder setup in depth. I believe that indexer acknowledgements are independent of SSL configuration and not related in this case.

The whole presentation is definitely worth reading (and watching, the recording is here: https://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4

View solution in original post

cantrell
New Member

The problem has less to due with self-signed certificates and far more to do with weak default algorithms...

0 Karma

jamesarmitage
Path Finder

Take a look at this presentation from .conf 2015:

https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

Page 6 has a nice exchange / function matrix, if you scan the "Forwarding" line you'll see that adding an SSL configuration stanza to a forwarder adds encryption to the data being sent to the indexers, as well as certificate authentication and CN checking.

See pages 7 and 8 for an attack scenario on an unsecured forwarder (provided the REST API is enabled on the forwarder). Pages 18-20 go over the forwarder setup in depth. I believe that indexer acknowledgements are independent of SSL configuration and not related in this case.

The whole presentation is definitely worth reading (and watching, the recording is here: https://conf.splunk.com/session/2015/recordings/2015-splunk-115.mp4

mlorch
Path Finder

Hi James, hi Duane,
Thanks for your answers and the links to this extremely helpful material! I will work though that.
In my opinion the “throwaway certificate” understanding is really important. So until now we had secured forwarder – indexer communication, but it was some kind of “security by obscurity”. I would appreciate a more detailed documentation by Splunk in the official documentation
Thanks again!

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Can we be more precise in your definition of "secured". Turning on SSL at all provides some level of "secure" (confidentiality) against casual snooping across the wire - even if you're using self-signed certificates on each end. If this is all you care about, then you've done well.

Next, the question of authentication comes up. Do the forwarders need to confirm the indexer they are talking to is "authentic"? If so, then SSL server authentication via certificate becomes needed. The forwarders need a copy of the CA cert chain signing the indexer's cert so they can verify it.

Continuing on authentication, does the indexer need to confirm the clients sending data to it are "authentic"? If so, the SSL client authentication becomes needed. With a throwaway certificate, it's hard to tell which forwarder is who, the throwaway cert becomes a shibboleth of sorts -- is the forwarder "one of us" or not? If you need to be able to tell forwarder A from forwarder B, then you need a unique certificate for each of them - and you need to be able to figure out how to handle revocation and stuff. (It gets complex)

0 Karma

gn694
Communicator

From watching the recording of your presentation I understand how using the default certificates can lead to authentication issues because of the root CA certificate being the same with every instance of Splunk.

If I do am not concerned with authentication, but want to ensure everything is encrypted, are the default certificates ok to use? Or would using our private CA (with one certificate shared used on all Universal Forwarders) provide some added benefit in terms of data encryption over the default certificates?

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Thanks James! Appreciate the positive comments on the talk. The main reason a forwarder needs a certificate is because of how the code parsing outputs.conf is written. There is no specific "SSL on or off" flag in outputs.conf - the present of the sslCertPath option in outputs carries the dual semantic of "Turn on SSL" and "Use this client certificate". It's a bit convoluted.

This reason is why in the SSL talk we use a single "throwaway" certificate on all of the forwarders, just to give them the option of turning SSL on without having to deal with managing unique certs and other PKI madness for potentially thousands of systems.

spl_unker
Explorer

Hi @dwaddle  and others

Im stuck with couple of questions and this thread is almost close to answer it.  Im actually working on securing communication between Splunk nodes.

I have 4 forwarders sending data to 3 indexers which is in cluster . I have a 1 deployment server to manage  forwarders . and have 1 cluster master:

Question 1 (securing forwarder->indexers): From the Splunk documentation and this thread i understand we need to have certificates for both forwarders and indexers. But don't understand why it is required on forwarders as well? Having certificates only in indexers does the job right? . Is it because of Splunk configuration or code demands to have it in forwarders as well?

Question 2 (Cluster Master->indexers) : Since we have already in production , is there any impact or precautions that needs to be taken while making communication secured  between CM and indexers. please share any link to document on the configuration . 

 

Thanks in Advance

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...