How do I set up SSL forwarding with new, self-signed certificates and authentication?

[hexx]

I would like to set up my Splunk-to-Splunk (forwarder to indexer) connections to use SSL with common-name-based authentication for my indexers, using self-signed server certificates created from a newly-created root certificate.

Which steps do I need to take to configure Splunk to accomplish this task?

[SandzVG]

Can someone at Splunk do another step-by-step for this with OWN Certificates , without changing reference names in each link for certificate names and making things crazy for such a simple requirement?

[hexx · Accepted solution]

Here is a detailed procedure to use non-default (in this case, self-signed) SSL certificates with common-name-based authentication (for the indexer(s) only) in a splunk2splunk (indexer to forwarder) connection.

1 - On a Splunk instance, create your own root certificate :

(As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate...)

NOTE : For clarity's sake, it is better to generate new certificates in another directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so.

# mkdir $SPLUNK_HOME/etc/certs

Point openssl to Splunk's openssl.cnf :

# export OPENSSL_CONF=$SPLUNK_HOME/openssl/openssl.cnf

Generate a new root certificate :

# $SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/certs

=> This will create a new certificate authority public certificate in $SPLUNK_HOME/etc/certs/cacert.pem This public CA certificate is to be distributed to all Splunk instances (indexers and forwarders) who will be checking server certificates signed with the root certificate we just generated (ca.pem).

2 - Generate a new self-signed server certificate for your indexer, specifying your indexer's host name as the common name recorded in the certificate :

(As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate...)

In our example, let's assume that your indexer's host name is "splunk-idx-01.example.com". Let's also assume that you want to use "changeme" as the password for your indexer's server certificate.

# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n splunk-idx-01 -c splunk-idx-01.example.com -p

• Enter "changeme" as the PEM pass phrase.
• Enter "splunk-idx-01.example.com" as the common name

• All other values are optional.

  => This will create a new server certificate in $SPLUNK_HOME/etc/certs/splunk-idx-01.pem. We will copy this server certificate to the indexer "splunk-idx-01.example.com, who will present it to the forwarders. The forwarders will use the public CA certificate from step 1 (cacert.pem) to check the identity of the indexer.

3 - Generate a new self-signed server certificate for your forwarders :

In our example, let's assume that you want to use "changeme2" as the password for your forwarders' server certificates.

# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n forwarder -p

• Enter "changeme2" as the PEM pass phrase.

• All other values are optional.

  => This will create a new server certificate in $SPLUNK_HOME/etc/certs/forwarder.pem. We will propagate this server certificate to all of our forwarders, who will present it to the indexer(s) when a connection is established. The indexer(s) will use the public CA certificate we generated in step 1 (cacert.pem) to check the credentials of the forwarders.

4 - Set up the indexer to use the newly created server certificate and to check the forwarders' certificates :

First, copy the $SPLUNK_HOME/etc/certs/splunk-idx-01.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your indexer and put them in a newly created $SPLUNK_HOME/etc/certs directory.

We will assume here that you will be using port 9997 to receive data from your forwarders.

In $SPLUNK_HOME/etc/system/local/inputs.conf, set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/certs/cacert.pem
serverCert = $SPLUNK_HOME/etc/certs/splunk-idx-01.pem
password = changeme
requireClientCert = false

[splunktcp-ssl:9997]
compressed = true

Important note regarding requireClientCert: As of Splunk 4.2.4, setting requireClientCert = true in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false".

Setting "requireClientCert = true" would require the following conditions to be met :

• "rootCA" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1.
• The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 5) is signed by that CA.

• The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 5). This password is "changeme2" in our example.

  The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.

  Restart Splunk after making these changes. Note that this will hash the certificate password in inputs.conf.

# $SPLUNK_HOME/bin/splunk restart

5 - Set up your forwarder(s) to use the newly created server certificates and to check the indexer's certificate with common name authentication :

Copy the $SPLUNK_HOME/etc/certs/forwarder.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your forwarder(s) and put them in a newly created $SPLUNK_HOME/etc/certs directory.

Then, define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf. Let's assume that your indexer's IP address is 192.168.1.100 :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 192.168.1.100:9997
compressed = true

[tcpout-server://192.168.1.100:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem
sslPassword = changeme2
sslVerifyServerCert = true
sslCommonNameToCheck = splunk-idx-01.example.com
altCommonNameToCheck = splunk-idx-01

Note that we have set "sslVerifyServerCert = true". This requires the following conditions to be met :

• "sslRootCAPath" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1.
• The indexer's "serverCert" (as defined in inputs.conf on step 4) is signed by that CA.
• The indexer has the password to read his own certificate ("password" in inputs.conf, as defined in step 4). This password is "changeme" in our example.

• The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 4) matches one of the "sslCommonNameToCheck" settings.

  This setup ensures that the client/forwarder only forwards to someone with a certificate signed by the CA and that that certificate was issued to someone with the expected name, i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file and the name signed into the indexer's certificate.

  If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate.

  Finally, restart the forwarder :

# $SPLUNK_HOME/bin/splunk restart

6 - You should now see the indexer and the forwarder establishing an SSL connection. Here's what it looks like in $SPLUNK_HOME/var/log/splunk/splunkd.log on both sides.

• on the indexer (with category.TcpInputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) :

(During start up...)

09-21-2010 18:47:29.735 DEBUG TcpInputProc - Initializing
09-21-2010 18:47:29.735 INFO  TcpInputProc - using queueSize 1000
09-21-2010 18:47:29.735 DEBUG TcpInputProc - creating tcp pipelineData queue
09-21-2010 18:47:29.735 DEBUG TcpInputProc - Reconfiguring
09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - clearing maps
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop enables2sHeartbeat=true
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop inputShutdownTimeout=90
09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - scanning configs
09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/splunk-idx-01.pem
09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/cacert.pem
09-21-2010 18:47:29.737 INFO  TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-21-2010 18:47:29.737 INFO  TcpInputProc - supporting SSL v2/v3
09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL dhfile=
09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL requireClientCert=1
09-21-2010 18:47:29.740 INFO  TcpInputProc - port 61025 is reserved for splunk 2 splunk (SSL)
09-21-2010 18:47:29.740 INFO  TcpInputProc - Port 61025 is compressed
09-21-2010 18:47:29.740 DEBUG TcpInputProc - readConfig - creating acceptor for port 61025
09-21-2010 18:47:29.740 DEBUG TcpInputProc - Initing Acceptor with  SSL
09-21-2010 18:47:29.740 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections

(...then when the forwarder connects)

09-21-2010 18:51:27.812 INFO  TcpInputProc - Connection in cooked mode from splunk-fwd-01.example.com
09-21-2010 18:51:27.816 INFO  TcpInputProc - Valid signature found
09-21-2010 18:51:27.816 INFO  TcpInputProc - Connection accepted from splunk-fwd-01.example.com

• on the forwarder (with category.TcpOutputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) :

(During start up...)

09-22-2010 16:22:12.017 DEBUG TcpOutputProc - groupName=[splunkssl] server=[192.168.1.100:9997]
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - creating group mapping for group splunkssl
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validation complete
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : initialBackoff=2
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxNumberOfRetriesAtHighestBackoff=-1
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxBackoff=20
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : backoffAtStartup=5
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will retry at max backoff sleep forever
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop maxNumberOfRetriesAtHighestBackoff=-1
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Using SSL for server 10.1.12.1:61025, sslCertPath=/opt/splunk/etc/certs/forwarder.pem
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Key file password requires decrypting
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslRootCAPath=/opt/splunk/etc/certs/cacert.pem
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslVerifyServerCert=1
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will verify Server's Certificate
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslCommonNameToCheck=splunk-idx-01.example.com
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will check server's Common Name against splunk-idx-01.example.com
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslAltNameToCheck=splunk-idx-01
09-22-2010 16:22:12.017 INFO  TcpOutputProc - ALL Connections will use SSL with sslCipher=
09-22-2010 16:22:12.018 INFO  TcpOutputProc - initializing single connection with retry strategy for 192.168.1.100:9997
09-22-2010 16:22:12.018 DEBUG TcpOutputProc - creating sender thread for group ssltest
09-22-2010 16:22:12.018 DEBUG TcpOutputProc - registering timer callback
09-22-2010 16:22:12.019 DEBUG TcpOutputProc - IndexAndForward explicitly set to : false
09-22-2010 16:22:12.019 DEBUG TcpOutputProc - returning from updateConfig
09-22-2010 16:22:12.020 INFO  TcpOutputProc - attempting to connect to 192.168.1.100:9997...
09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997
09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validation complete
09-22-2010 16:22:12.029 INFO  TcpOutputProc - Connected to 192.168.1.100:9997 
09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Sending signature
09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Signature sent

(...then, as the forwarder keeps the connection alive about every 30 seconds...)

09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Inserting timer token into the queue.
09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Processing timer event from the queue.
09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Size of sockets is: 1
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Inserting timer token into the queue.
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Processing timer event from the queue.
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Size of sockets is: 1

7 - Troubleshooting : If things don't work out right!

a) First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".

b) If necessary, increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.

On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect.

c) Check the SSL configuration as it is seen by Splunk using btool.

• On the indexer :

$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug

• On the forwarder :

$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

d) Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :

• The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error :

12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory

• The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error :

12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :

# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :

C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text

e) More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :

http://www.splunk.com/base/Documentation/latest/Admin/EncryptandauthenticatedatawithSSL http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL

The Splunk community wiki also has detailed tutorials on how to set up SSL for forwarding with 3 different scenarios :

• Configuring Splunk forwarding to use the default SSL server certificate
• Configuring Splunk forwarding to use SSL certificates self-signed by a newly generated root certific...
• Configuring Splunk forwarding to use SSL certificates signed by a third party Certificate Authority

[jpg0002]

Note: Step 2: Splunk 8.0.1 distribution contains a deprecated version of genSignedServerCert.sh. When run, genSignedServerCert.sh says to use "splunk createssl server-cert" instead.

[jpg0002]

Also, a couple of updates:
in Step 4, for the indexer's inputs.conf, according to splunkd.log password and rootCA are deprecated; we should instead use sslPassword and sslRootCAPath respectively.

[gauravsplunkarc]

Now that I need to offload SSL to F5 VIP, what changes do I need to make on universal forwarders?

[inventsekar]

Thanks Hexx for the great post.

for version 6.6.1, the document https://docs.splunk.com/Documentation/Splunk/6.6.1/Security/ConfigureSplunkforwardingtousesignedcert...

says, on indexer, we need to update inputs.conf and server.conf as well.
same for forwarders as well. the above steps say only about inputs.conf file, not server.conf (which was true for older versions)

[jrodman]

requireClientCert currently does not interoperate successfully with splunkweb, the cli rest client, and possibly deployment client. Much of this is being worked on presently, not for 4.2.4 but for a bit later.

[hexx]

@mfeeny1 : I fear that this is the same issue mentioned in the tutorial where "requireClientCert = true" in the indexer's inputs.conf causes the SSL communication to fail, regardless of how valid the certificates are. This is bug SPL-37637. Please open a support case if you want to have this assessed further and tracked.

[mfeeny1]

hexx,

Thx for the response! Little by little, I am figuring this stuff out (I think!! 😉

Here is where I am now... I have Forwarder-Indexer SSL communication WORKING - it is using our "corporate" cert (not Splunk's), and it is also doing Authentication.

BUT... (always a but, right)... Although I feel 99% sure I am using comparable settings, I can NOT get Deployment Client - Deployment Server SSL communications to work.

Actually... If I do NOT request Authentication of the Client, then it DOES work, with the corporate certs. But, if I request Authentication ("requireClientCert = true", specified in Deployment Server's server.conf), it fails. When it fails, the Deployment Server's splunkd.log has an entry that says:

TcpInputProc - SSL clause not found or servercert not provided - SSL ports will not be available

On the Deployment Client, a packet capture shows that the Deployment Server sent the following SSL message, during the SSL handshake:

Alert (Level: Fatal, Description: Bad Certificate)

SO... As part of my troubleshooting, I opened up server.conf.spec, to tease out whatever info I could, regarding the "requireClientCert" attribute. Here is what it says:

• Used by distributed deployment: The deployment server requires that deployment clients are authenticated before allowing them to poll for new configurations/applications. * If true, a client can connect ONLY if a certificate created by our certificate authority was used on that client.

(Note: The bolding of "our", above, is mine, not server.conf.spec's.) Am I reading to closely if I interpret our to mean Splunk's, and not our corporate CA??? Is it possible that this is failing because it doesn't see Splunk's CA in my client cert?? This seems illogical, but I'm running out of logic, at this point 😞

It also seems illogical, because Forwarder-Indexer SSL communication is working, and I'm using the same Client cert for that traffic.

Bottom line... I'll accept any ideas anyone has, on what I'm doing wrong, or how I can diagnose this problem?

Thx MUCH!

mfeeny1

[hexx]

@mfeeny1 : If all of your indexers have certificates signed by the same CA (which I imagine is the case), you'll simply need to make sure that they each have their own [tcpout-server://server:port] stanza and that "sslRootCAPath" in each of these stanzas points to a file containing the public key of the CA that signed the server certificates present on the indexers. sslCertPath does indeed refer to the certificate presented by the forwarder to the indexer, but note that due to a bug, that certificate cannot currently be checked by the indexer. Regardless of that, it still needs to be present!

[mfeeny1]

Hello...

I am about to configure SSL for Forwarder-to-Indexer communication, so this post is a Gift from Above! I did have one question, though, regarding the following piece of information...

"If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate"

As has been pointed out by ziegfried above, the config option is "sslCertPath" (no "Server"). But, that's not what confuses me. What got me is... The sslCertPath option is specified on the Forwarder, and refers to the forwarder/client cert, right??? The forwarder does not reference the Indexer/Server cert, right??

Thx for any clarification for this rookie 🙂

mfeeny1

[hexx]

Thank you for pointing it out Ziegfrid, and to Araitz for fixing it!

[sdkp03]

Hey Hex, I have similar issue but in my case its the communication between the license master and the peers. Can I still use same steps to generate self signed certs?

[ziegfried]

There is a mistake in your example for configuration on the forwarder side. The config option is called sslCertPath and not sslServerCertPath. Please correct that...