How to secure the Splunk platform with SSL

[BRFZ]

Hello,

I have a distributed Splunk architecture with a single search head, two indexers, and management tier : License Master, Monitoring Console, and Deployment Server, in addition to the forwarders. SSL has already been configured for the web interfaces, but I would now like to secure the remaining components and establish SSL-encrypted connections between them as well.

The certificates we are using are self-generated. Could you please guide me on how to proceed with securing all internal communications in this setup? Specifically, I would like to know if I should auto-generate a new certificate for each component and each connection or if there’s an efficient way to manage SSL across the entire environment.

Thank you in advance for your help!

[BRFZ]

Hi Dear Community,

I am encountering the following error across all servers:

SSLCommon - Can't read key file from /opt/splunk/etc/auth/CERT.pem

[isoutamo]

Have you checked that this file exists and your splunk user have read access to it?

[BRFZ]

Yes, I have verified it, and everything is correct.

[isoutamo]

Sorry I didn’t read correctly the error message. It said that splunk cannot read key file from your pem file. Are you sure that it contains all needed parts inside it?

[BRFZ]

The format of a .pem file is as follows: 

-----BEGIN CERTIFICATE-----

-----END CERTIFICATE-----

[isoutamo]

Have you read and understand what this presentation said https://conf.splunk.com/files/2023/slides/SEC1936B.pdf ?
There is also video presentation about it. Those should explain how this should do.

[BRFZ]

From what I understand, I need to combine the .pem file with the private key, and this combined file is what I should use in the configuration, correct ? 

[isoutamo]

Can you show your conf files and explain what you have in which pem files?
Please hide real passwords etc.

[BRFZ]

Here is the configuration file. 

[sslConfig]
enableSplunkdSSL = true
sslRootCAPath = /opt/splunk/etc/auth/cert/CA.pem
serverCert = /opt/splunk/etc/auth/cert/srv.pem

For the PEM files, as mentioned earlier, they contain the 'BEGIN CERTIFICATE' and 'END CERTIFICATE' sections.

[isoutamo]

I expecting that this is your server.conf file?

As you are using your private CA you must add those chains into serverCert pem file.

You can read more about it from https://docs.splunk.com/Documentation/Splunk/latest/Security/HowtoprepareyoursignedcertificatesforSp... or that conf presentation or any other TLS cert documentation.

Base on your description you haven't done this for your serverCert pem file.

e.g I have this in one of my conf file (maybe not exactly the same what you will need)

-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

 You should have also that RSA PRIVATE KEY in your pem file and also add parameter for it's password into your server.conf.

[BRFZ]

Thank you so much for your help. I am pleased to share that I was able to resolve the initial issue by adjusting the PEM file.

However, when I execute the command: openssl s_client -showcerts -connect hostname:port I get a connected status, but it ultimately results in the following error: 80FB2563307F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:317:

Additionally, another error is displayed:
Verification error: self-signed certificate in certificate chain

Your help would be greatly appreciated.

[PickleRick]

Let me quote myself from earlier:

"Just one important thing - if you want to enable TLS, get yourself a CA and issue proper certificates. Using self-signeds everywhere will not help you much securitywise and you'll run into troubles when trying to validate them properly (which might be your case)".

If you created self-signed certs for your components, you will have problems validating them. If you have a CA from which you issued those certs, you've probably not configured the root CA's cert as trusted.

[BRFZ]

I have followed the configuration steps as outlined, but unfortunately, I have lost the connection between the components. I have applied the certificate configurations and other related settings in server.conf, including modifying the search peers to use HTTPS in distsearch.conf.

I also modified the master license slave to use HTTPS, but I did not make any changes to the license master itself. Could you confirm if there are any specific configurations required on the master license?

After applying the changes, the Server Manager has become extremely slow, and I can no longer access its web interface. Additionnaly, I lost connectivity between the components. 

Is there someone who could help me with resolving this issue please?

[PickleRick]

It's hard to tell you how to fix your setup when we don't know the details of your configuration and your certs.

Just one important thing - if you want to enable TLS, get yourself a CA and issue proper certificates. Using self-signeds everywhere will not help you much securitywise and you'll run into troubles when trying to validate them properly (which might be your case)

[dural_yyz]

If your indexers and other devices are no longer indexing data then you need to check individual server splunkd.log files.  Tail and grep for details around connections.

Any error codes will help you and us in determining the issues.

[isoutamo]

This explains things more easily than those docs if you haven’t earlier experience about TLS https://conf.splunk.com/files/2023/slides/SEC1936B.pdf

[inventsekar]

Hi @BRFZ 

(As others have not mentioned it yet) maybe pls have a look at this doc,.. it got pretty good details:

https://docs.splunk.com/Documentation/Splunk/9.4.0/Security/WhatyoucansecurewithSplunk

[BRFZ]

Thank you for your response and the provided documentation.
I’ve already followed the steps, but encountered communication issues and I had to reset the configuration in order to restore connectivity.

Could you please provide a more detailed procedure or tailored guidance for my case to help me securely configure TLS/SSL?

[dural_yyz]

https://lantern.splunk.com/Splunk_Platform/Product_Tips/Administration/Securing_the_Splunk_platform_...

These articles can explain it much better than I can and it is coming straight from the source.

[BRFZ]

Thank you @dural_yyz for your prompt response and for providing the documentation. However, I need further assistance regarding the SSL certificates that need to be generated for my Splunk environment.

Could you please clarify whether I need to generate a separate certificate for each component (e.g., search head, indexers, forwarders, etc.)? Additionally, do I need to create different certificates for the various connections between these components?