topic Re: How to secure the Splunk platform with SSL in Deployment Architecture

How to secure the Splunk platform with SSL

BRFZ — Tue, 12 Nov 2024 13:21:13 GMT

Hello,

I have a distributed Splunk architecture with a single search head, two indexers, and management tier : License Master, Monitoring Console, and Deployment Server, in addition to the forwarders. SSL has already been configured for the web interfaces, but I would now like to secure the remaining components and establish SSL-encrypted connections between them as well.

The certificates we are using are self-generated. Could you please guide me on how to proceed with securing all internal communications in this setup? Specifically, I would like to know if I should auto-generate a new certificate for each component and each connection or if there’s an efficient way to manage SSL across the entire environment.

Thank you in advance for your help!

Re: How to secure the Splunk platform with SSL

dural_yyz — Tue, 12 Nov 2024 14:07:29 GMT

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Securing_the_Splunk_platform_with_TLS

These articles can explain it much better than I can and it is coming straight from the source.

Re: How to secure the Splunk platform with SSL

BRFZ — Tue, 12 Nov 2024 14:42:55 GMT

Thank you @dural_yyz for your prompt response and for providing the documentation. However, I need further assistance regarding the SSL certificates that need to be generated for my Splunk environment.

Could you please clarify whether I need to generate a separate certificate for each component (e.g., search head, indexers, forwarders, etc.)? Additionally, do I need to create different certificates for the various connections between these components?

Re: How to secure the Splunk platform with SSL

PickleRick — Tue, 12 Nov 2024 15:03:43 GMT

As a general rule, you should _always_ create separate certificates for separate entities (in your case - for separate components).

Also remember that if you decide to enable client authentication, certificate must be issued with proper key usage.

Re: How to secure the Splunk platform with SSL

BRFZ — Wed, 18 Dec 2024 10:30:36 GMT

Thank you for your response and the provided documentation.
I’ve already followed the steps, but encountered communication issues and I had to reset the configuration in order to restore connectivity.

Could you please provide a more detailed procedure or tailored guidance for my case to help me securely configure TLS/SSL?

Re: How to secure the Splunk platform with SSL

inventsekar — Wed, 18 Dec 2024 13:20:17 GMT

Hi @BRFZ

(As others have not mentioned it yet) maybe pls have a look at this doc,.. it got pretty good details:

https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/WhatyoucansecurewithSplunk

Re: How to secure the Splunk platform with SSL

isoutamo — Wed, 18 Dec 2024 18:12:25 GMT

This explains things more easily than those docs if you haven’t earlier experience about TLS https://conf.splunk.com/files/2023/slides/SEC1936B.pdf

Re: How to secure the Splunk platform with SSL

BRFZ — Tue, 24 Dec 2024 13:53:56 GMT

I have followed the configuration steps as outlined, but unfortunately, I have lost the connection between the components. I have applied the certificate configurations and other related settings in server.conf, including modifying the search peers to use HTTPS in distsearch.conf.

I also modified the master license slave to use HTTPS, but I did not make any changes to the license master itself. Could you confirm if there are any specific configurations required on the master license?

After applying the changes, the Server Manager has become extremely slow, and I can no longer access its web interface. Additionnaly, I lost connectivity between the components.

Is there someone who could help me with resolving this issue please?

Re: How to secure the Splunk platform with SSL

dural_yyz — Tue, 24 Dec 2024 16:05:34 GMT

If your indexers and other devices are no longer indexing data then you need to check individual server splunkd.log files. Tail and grep for details around connections.

Any error codes will help you and us in determining the issues.

Re: How to secure the Splunk platform with SSL

PickleRick — Wed, 25 Dec 2024 11:45:19 GMT

It's hard to tell you how to fix your setup when we don't know the details of your configuration and your certs.

Just one important thing - if you want to enable TLS, get yourself a CA and issue proper certificates. Using self-signeds everywhere will not help you much securitywise and you'll run into troubles when trying to validate them properly (which might be your case)

Re: How to secure the Splunk platform with SSL

BRFZ — Mon, 30 Dec 2024 15:23:28 GMT

Hi Dear Community,

I am encountering the following error across all servers:

SSLCommon - Can't read key file from /opt/splunk/etc/auth/CERT.pem

Re: How to secure the Splunk platform with SSL

isoutamo — Mon, 30 Dec 2024 15:33:43 GMT

Have you checked that this file exists and your splunk user have read access to it?

Re: How to secure the Splunk platform with SSL

BRFZ — Mon, 30 Dec 2024 15:38:38 GMT

Yes, I have verified it, and everything is correct.

Re: How to secure the Splunk platform with SSL

isoutamo — Mon, 30 Dec 2024 16:58:23 GMT

Sorry I didn’t read correctly the error message. It said that splunk cannot read key file from your pem file. Are you sure that it contains all needed parts inside it?

Re: How to secure the Splunk platform with SSL

BRFZ — Tue, 31 Dec 2024 08:39:52 GMT

The format of a .pem file is as follows:

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

Re: How to secure the Splunk platform with SSL

isoutamo — Tue, 31 Dec 2024 08:47:05 GMT

Have you read and understand what this presentation said https://conf.splunk.com/files/2023/slides/SEC1936B.pdf ?
There is also video presentation about it. Those should explain how this should do.

Re: How to secure the Splunk platform with SSL

BRFZ — Tue, 31 Dec 2024 08:58:51 GMT

From what I understand, I need to combine the .pem file with the private key, and this combined file is what I should use in the configuration, correct ?

Re: How to secure the Splunk platform with SSL

isoutamo — Tue, 31 Dec 2024 09:10:53 GMT

Can you show your conf files and explain what you have in which pem files?
Please hide real passwords etc.

Re: How to secure the Splunk platform with SSL

BRFZ — Tue, 31 Dec 2024 15:03:33 GMT

Here is the configuration file.

[sslConfig] enableSplunkdSSL = true sslRootCAPath = /opt/splunk/etc/auth/cert/CA.pem serverCert = /opt/splunk/etc/auth/cert/srv.pem

For the PEM files, as mentioned earlier, they contain the 'BEGIN CERTIFICATE' and 'END CERTIFICATE' sections.

Re: How to secure the Splunk platform with SSL

isoutamo — Tue, 31 Dec 2024 10:18:23 GMT

I expecting that this is your server.conf file?

As you are using your private CA you must add those chains into serverCert pem file.

You can read more about it from https://docs.splunk.com/Documentation/Splunk/latest/Security/HowtoprepareyoursignedcertificatesforSplunk or that conf presentation or any other TLS cert documentation.

Base on your description you haven't done this for your serverCert pem file.

e.g I have this in one of my conf file (maybe not exactly the same what you will need)

You should have also that RSA PRIVATE KEY in your pem file and also add parameter for it's password into your server.conf.