I would like to set up my Splunk-to-Splunk (forwarder to indexer) connections to use SSL with common-name-based authentication for my indexers, using self-signed server certificates created from a newly-created root certificate.
Which steps do I need to take to configure Splunk to accomplish this task?
Here is a detailed procedure to use non-default (in this case, self-signed) SSL certificates with common-name-based authentication (for the indexer(s) only) in a splunk2splunk (indexer to forwarder) connection.
NOTE : For clarity's sake, it is better to generate new certificates in another directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so.
# mkdir $SPLUNK_HOME/etc/certs
Point openssl to Splunk's openssl.cnf :
# export OPENSSL_CONF=$SPLUNK_HOME/openssl/openssl.cnf
Generate a new root certificate :
# $SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/certs
=> This will create a new certificate authority public certificate in $SPLUNK_HOME/etc/certs/cacert.pem This public CA certificate is to be distributed to all Splunk instances (indexers and forwarders) who will be checking server certificates signed with the root certificate we just generated (ca.pem).
In our example, let's assume that your indexer's host name is "splunk-idx-01.example.com". Let's also assume that you want to use "changeme" as the password for your indexer's server certificate.
# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n splunk-idx-01 -c splunk-idx-01.example.com -p
All other values are optional.
=> This will create a new server certificate in $SPLUNK_HOME/etc/certs/splunk-idx-01.pem. We will copy this server certificate to the indexer "splunk-idx-01.example.com, who will present it to the forwarders. The forwarders will use the public CA certificate from step 1 (cacert.pem) to check the identity of the indexer.
In our example, let's assume that you want to use "changeme2" as the password for your forwarders' server certificates.
# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n forwarder -p
All other values are optional.
=> This will create a new server certificate in $SPLUNK_HOME/etc/certs/forwarder.pem. We will propagate this server certificate to all of our forwarders, who will present it to the indexer(s) when a connection is established. The indexer(s) will use the public CA certificate we generated in step 1 (cacert.pem) to check the credentials of the forwarders.
First, copy the $SPLUNK_HOME/etc/certs/splunk-idx-01.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your indexer and put them in a newly created $SPLUNK_HOME/etc/certs directory.
We will assume here that you will be using port 9997 to receive data from your forwarders.
In $SPLUNK_HOME/etc/system/local/inputs.conf, set up the following stanzas :
[SSL] rootCA = $SPLUNK_HOME/etc/certs/cacert.pem serverCert = $SPLUNK_HOME/etc/certs/splunk-idx-01.pem password = changeme requireClientCert = false [splunktcp-ssl:9997] compressed = true
Important note regarding
requireClientCert: As of Splunk 4.2.4, setting
requireClientCert = true in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep
requireClientCert set to "false".
Setting "requireClientCert = true" would require the following conditions to be met :
The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 5). This password is "changeme2" in our example.
The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.
Restart Splunk after making these changes. Note that this will hash the certificate password in inputs.conf.
# $SPLUNK_HOME/bin/splunk restart
Copy the $SPLUNK_HOME/etc/certs/forwarder.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your forwarder(s) and put them in a newly created $SPLUNK_HOME/etc/certs directory.
Then, define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf. Let's assume that your indexer's IP address is 192.168.1.100 :
[tcpout] defaultGroup = splunkssl [tcpout:splunkssl] server = 192.168.1.100:9997 compressed = true [tcpout-server://192.168.1.100:9997] sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem sslPassword = changeme2 sslVerifyServerCert = true sslCommonNameToCheck = splunk-idx-01.example.com altCommonNameToCheck = splunk-idx-01
Note that we have set "sslVerifyServerCert = true". This requires the following conditions to be met :
The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 4) matches one of the "sslCommonNameToCheck" settings.
This setup ensures that the client/forwarder only forwards to someone with a certificate signed by the CA and that that certificate was issued to someone with the expected name, i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file and the name signed into the indexer's certificate.
If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate.
Finally, restart the forwarder :
# $SPLUNK_HOME/bin/splunk restart
(During start up...)
09-21-2010 18:47:29.735 DEBUG TcpInputProc - Initializing 09-21-2010 18:47:29.735 INFO TcpInputProc - using queueSize 1000 09-21-2010 18:47:29.735 DEBUG TcpInputProc - creating tcp pipelineData queue 09-21-2010 18:47:29.735 DEBUG TcpInputProc - Reconfiguring 09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - clearing maps 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop enables2sHeartbeat=true 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600 09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop inputShutdownTimeout=90 09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - scanning configs 09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/splunk-idx-01.pem 09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/cacert.pem 09-21-2010 18:47:29.737 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM 09-21-2010 18:47:29.737 INFO TcpInputProc - supporting SSL v2/v3 09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL dhfile= 09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL requireClientCert=1 09-21-2010 18:47:29.740 INFO TcpInputProc - port 61025 is reserved for splunk 2 splunk (SSL) 09-21-2010 18:47:29.740 INFO TcpInputProc - Port 61025 is compressed 09-21-2010 18:47:29.740 DEBUG TcpInputProc - readConfig - creating acceptor for port 61025 09-21-2010 18:47:29.740 DEBUG TcpInputProc - Initing Acceptor with SSL 09-21-2010 18:47:29.740 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
(...then when the forwarder connects)
09-21-2010 18:51:27.812 INFO TcpInputProc - Connection in cooked mode from splunk-fwd-01.example.com 09-21-2010 18:51:27.816 INFO TcpInputProc - Valid signature found 09-21-2010 18:51:27.816 INFO TcpInputProc - Connection accepted from splunk-fwd-01.example.com
(During start up...)
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - groupName=[splunkssl] server=[192.168.1.100:9997] 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - creating group mapping for group splunkssl 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validation complete 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : initialBackoff=2 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxNumberOfRetriesAtHighestBackoff=-1 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxBackoff=20 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : backoffAtStartup=5 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will retry at max backoff sleep forever 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop maxNumberOfRetriesAtHighestBackoff=-1 09-22-2010 16:22:12.017 INFO TcpOutputProc - Using SSL for server 10.1.12.1:61025, sslCertPath=/opt/splunk/etc/certs/forwarder.pem 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Key file password requires decrypting 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslRootCAPath=/opt/splunk/etc/certs/cacert.pem 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslVerifyServerCert=1 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will verify Server's Certificate 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslCommonNameToCheck=splunk-idx-01.example.com 09-22-2010 16:22:12.017 INFO TcpOutputProc - Will check server's Common Name against splunk-idx-01.example.com 09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslAltNameToCheck=splunk-idx-01 09-22-2010 16:22:12.017 INFO TcpOutputProc - ALL Connections will use SSL with sslCipher= 09-22-2010 16:22:12.018 INFO TcpOutputProc - initializing single connection with retry strategy for 192.168.1.100:9997 09-22-2010 16:22:12.018 DEBUG TcpOutputProc - creating sender thread for group ssltest 09-22-2010 16:22:12.018 DEBUG TcpOutputProc - registering timer callback 09-22-2010 16:22:12.019 DEBUG TcpOutputProc - IndexAndForward explicitly set to : false 09-22-2010 16:22:12.019 DEBUG TcpOutputProc - returning from updateConfig 09-22-2010 16:22:12.020 INFO TcpOutputProc - attempting to connect to 192.168.1.100:9997... 09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997 09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validation complete 09-22-2010 16:22:12.029 INFO TcpOutputProc - Connected to 192.168.1.100:9997 09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Sending signature 09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Signature sent
(...then, as the forwarder keeps the connection alive about every 30 seconds...)
09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Inserting timer token into the queue. 09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Processing timer event from the queue. 09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Size of sockets is: 1 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Inserting timer token into the queue. 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Processing timer event from the queue. 09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Size of sockets is: 1
a) First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".
b) If necessary, increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.
On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect.
c) Check the SSL configuration as it is seen by Splunk using btool.
$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug
$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug
d) Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :
12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory
12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem
On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :
# openssl rsa -in /opt/splunk/etc/auth/server.pem -text
The same can be done on Windows with the openssl binary that ships with Splunk :
C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text
e) More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :
The Splunk community wiki also has detailed tutorials on how to set up SSL for forwarding with 3 different scenarios :
There is a mistake in your example for configuration on the forwarder side. The config option is called sslCertPath and not sslServerCertPath. Please correct that...
Thank you for pointing it out Ziegfrid, and to Araitz for fixing it!
I am about to configure SSL for Forwarder-to-Indexer communication, so this post is a Gift from Above! I did have one question, though, regarding the following piece of information...
"If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate"
As has been pointed out by ziegfried above, the config option is "sslCertPath" (no "Server"). But, that's not what confuses me. What got me is... The sslCertPath option is specified on the Forwarder, and refers to the forwarder/client cert, right??? The forwarder does not reference the Indexer/Server cert, right??
Thx for any clarification for this rookie 🙂
@mfeeny1 : If all of your indexers have certificates signed by the same CA (which I imagine is the case), you'll simply need to make sure that they each have their own [tcpout-server://server:port] stanza and that "sslRootCAPath" in each of these stanzas points to a file containing the public key of the CA that signed the server certificates present on the indexers. sslCertPath does indeed refer to the certificate presented by the forwarder to the indexer, but note that due to a bug, that certificate cannot currently be checked by the indexer. Regardless of that, it still needs to be present!
Thx for the response! Little by little, I am figuring this stuff out (I think!! 😉
Here is where I am now... I have Forwarder-Indexer SSL communication WORKING - it is using our "corporate" cert (not Splunk's), and it is also doing Authentication.
BUT... (always a but, right)... Although I feel 99% sure I am using comparable settings, I can NOT get Deployment Client - Deployment Server SSL communications to work.
Actually... If I do NOT request Authentication of the Client, then it DOES work, with the corporate certs. But, if I request Authentication ("requireClientCert = true", specified in Deployment Server's server.conf), it fails. When it fails, the Deployment Server's splunkd.log has an entry that says:
TcpInputProc - SSL clause not found or servercert not provided - SSL ports will not be available
On the Deployment Client, a packet capture shows that the Deployment Server sent the following SSL message, during the SSL handshake:
Alert (Level: Fatal, Description: Bad Certificate)
SO... As part of my troubleshooting, I opened up server.conf.spec, to tease out whatever info I could, regarding the "requireClientCert" attribute. Here is what it says:
(Note: The bolding of "our", above, is mine, not server.conf.spec's.) Am I reading to closely if I interpret our to mean Splunk's, and not our corporate CA??? Is it possible that this is failing because it doesn't see Splunk's CA in my client cert?? This seems illogical, but I'm running out of logic, at this point 😞
It also seems illogical, because Forwarder-Indexer SSL communication is working, and I'm using the same Client cert for that traffic.
Bottom line... I'll accept any ideas anyone has, on what I'm doing wrong, or how I can diagnose this problem?
@mfeeny1 : I fear that this is the same issue mentioned in the tutorial where "requireClientCert = true" in the indexer's inputs.conf causes the SSL communication to fail, regardless of how valid the certificates are. This is bug SPL-37637. Please open a support case if you want to have this assessed further and tracked.
requireClientCert currently does not interoperate successfully with splunkweb, the cli rest client, and possibly deployment client. Much of this is being worked on presently, not for 4.2.4 but for a bit later.
Thanks Hexx for the great post.
for version 6.6.1, the document https://docs.splunk.com/Documentation/Splunk/6.6.1/Security/ConfigureSplunkforwardingtousesignedcert...
says, on indexer, we need to update inputs.conf and server.conf as well.
same for forwarders as well. the above steps say only about inputs.conf file, not server.conf (which was true for older versions)