Setting up SSL in a Distributed deployment

[SwatiApte]

We have a distributed Splunk set up with SHC and Indexer Clustering enabled. Communication between a user browser and the SHC members is encrypted using TLS and Certificates. We are now trying to set up Universal forwarders on the source systems to acquire the data. I have been trying to read some material around this and came across few very useful links such as:
https://conf.splunk.com/session/2015/conf2015_DWaddle_DefensePointSecurity_deploying_SplunkSSLBestPr...

However, I still have the following questions and if you could provide some pointers around this, it will be very helpful:
1. From the documentation it appears that we need to generate one Server certificate per Indexer (assume we have 6 in our cluster) and enable SSL in the inputs.conf file pointing to the server/ca certs... On the forwarders though, should we have one certificate per forwarder (assume we have 10 UFs) or one common certificate across all of them ?
2. On the Forwarder's outputs.conf we need to set the sslCommonNameToCheck attribute to match the indexer server's certificate common name. What is the purpose of this attribute ? Why do we not have something similar on the indexer side ?
3. If 10 separate applications are forwarding data to our indexers - is our understanding correct that we just need to enable SSL on the indexer once with the certificate generated as per point 1 above and use "same" forwarder certificate across all forwarders or have 1 certificate per app ?
4. If we manage the Forwarders configurations via Deployment Server, is it a good practice to enable SSL for this communication as well ? Can we distribute the certificates automatically to forwarders this way and will the password get encrypted in such a case as we would be pushing app specific folders?

[jkat54 · Accepted solution]

1. You can use the same ssl certificate on each indexer or have different certs per indexer. The choice is yours. You specify the cert in outputs.conf stanzas for one stanza that references all the indexers or for each indexer in separate stanzas.

2. sslCommonNameToCheck works everywhere, not just on the forwarders.

3. You just need one forwarder ssl certificate for all forwarders. You could do one for each if you were crazy though.

4. The password will get encrypted everywhere but the deployment server. So it's best to use self signed here and have a cert renewal process defined if you deploy them via deployment server. I do it all the time though. If higher security is desired, deploy ssl certs to forwarders as part of the install process on the forwarders.

[SwatiApte]

Thanks jkat54 for the response. For point no.1, I understand now that we can set up 1 cert for all indexers or 1 per indexer and you have specified the choice is upto us. But are there any standard/best practices that we can look at to decide what choice to make ?
For point no. 4, is that a limitation within Splunk at the moment ? I will need to explore if we can use self-signed certificates along with Org CA certs to communicate with indexers.

[jkat54]

1. The option mainly exists so that you can use different ssl certs for different indexer clusters. In case you send the data to two different Splunk silos/teams for example. Perhaps infosec has one ssl cert, and operations uses another for example. In the end your security best practices are totally up to you. Having 20 certs just for indexers becomes a burden, but some organizations may deem that necessary. The de facto best practice is to use your own certs, regardless of how many you want to use... Just don't use the out of the box Splunk cert. this is especially important if your Splunk indexers exist on shared infrastructure such as "the cloud".

4 (four). I wouldn't call it a limitation so much but then again the ssl configuration of Splunk has been known to be a little more difficult that it probably should be. Honestly I appreciate the complexity because there's also a lot of flexibility. Many of the other apps out there don't even let you change the ssl certificates used, let alone specify so many different options and use a cert for each function etc. It's difficult to grasp which increases time required to deploy proper ssl, but at least you can do it properly!

[SwatiApte]

Hi jkat54, just a final question I had about this while setting it all up... Our understanding is that we can have indexers receive data from some forwarders over SSL port (let us say 9998) and from some forwarders over standard TCP port (say 9997). The question is once we set up SSL, it wont automatically apply to all existing forwarders where SSL is not set up and that they will continue to function as is.