Deployment Architecture

SSL is working for Forwarder-Indexer, but not Deployment Client-Server traffic

Path Finder

Hi. I have a Deployment Server, and I am using it to configure a Deployment Client, which is a Windows Universal Forwarder. I want to use SSL for both Forwarder-Indexer communications, as well as Deployment Client - Deployment Server communications, using "our" Certs (not Splunk's).

The initial state is that Forwarder-Indexer communication, as well as Deployment Client - Deployment Server communication, is working fine, but not using our Certs.

My first step is to configure the Forwarder-Indexer communication to use our Certs. So, I modify outputs.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. It works like a charm - the UF is now sending data to the Indexers, using SSL, with our Certs.

So now I want to make the equivalent change for the Deployment Client - Deployment Server communication. This time, I modify server.conf in the appropriate deployment-apps directory on the Deployment Server, with the needed SSL parameters, and reload the app. This does NOT work. splunkd.log, on both Deployment Client and Deployment Server, report errors (shown below), and the two endpoints no longer communicate, and a "./splunk list deploy-clients" command on the Deployment Server no longer displays the target machine as one of its Deployment Clients.

As best as I can tell, I am using the same "forwarder.pem", "cacert.pem", and cert passwords in the two .conf files (outputs and server), but things just aren't working for the Deployment Client - Deployment Server communications.

Any enlightenment, or at least tips on troubleshooting this, would be GREATLY appreciated!

Here are the splunkd.log errors...

On the Deployment Client

09-21-2011 20:38:02.971 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
09-21-2011 20:38:02.971 -0700 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig

09-21-2011 20:38:03.994 -0700 ERROR SSLCommon - Can't read key file C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
09-21-2011 20:38:03.994 -0700 ERROR HTTPServer - SSL will not be enabled

On the Deployment Server

09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error = error:1407609C:SSL routines:SSL23_GET_CLIENT_HELLO:http request
09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - ACCEPT_RESULT=-1 VERIFY_RESULT=0
09-21-2011 20:36:13.548 -0700 ERROR TcpInputFd - SSL Error for fd from HOST:*hostname*, IP:*ip-addr*, PORT:50252

And, here are the .conf files...

outputs.conf

[tcpout]  
defaultGroup = splunkssl-autolb-group  
disabled = false

[tcpout-server://*hostname*:9998]  
sslCertPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\forwarder.pem  
sslPassword=*password*  
sslRootCAPath=C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local\cacert.pem  
sslVerifyServerCert=true  
sslCommonNameToCheck=*full_hostname*  
altCommonNameToCheck=*short_hostname*  

there is more, but not SSL-related

server.conf

[sslConfig]  
enableSplunkdSSL = true  
sslKeysfile = forwarder.pem  
sslKeysfilePassword = *password*  
caCertFile = cacert.pem  
caPath = C:\Program Files\SplunkUniversalForwarder\etc\apps\\*app-name*\local  

Splunk Employee
Splunk Employee

I fear that the problem here is that on first time run, splunkd encodes the default password for the default server certificate used for the splunkd management port, and outputs that to $SPLUNK_HOME/etc/system/local/server.conf :

# cat etc/system/local/server.conf 
[sslConfig]
sslKeysfilePassword = $1$S6+KkTowacjF

As you know, settings in etc/system/local will take precedence over any other instance of the same configuration key anywhere else. Which means that the sslKeysfilePassword that you set with deployment server will not be taken into account.

You can correct this by deleting the sslKeysfilePassword entry in etc/system/local/server.conf and restarting splunkd, but this is not something that deployment client can do for you.

Path Finder

hexx, others,

This is EXACTLY what I concluded as well, after careful troubleshooting: My password value, in my app version of server.conf, was being trumped by the etc/system/local version.

So, I first tried fixing this with the following Install sequence...

1) Install via msiexec command, with the LAUNCHSPLUNK=0 flag, to prevent Splunk from starting and creating server.conf.

2) Create, and copy files to, my app directory, including a version of server.conf with MY certs and password.

3) Start Splunk.

Unfortunately, this did NOT work. I found that, even with the LAUNCHSPLUNK=0 flag, splunk creates etc/system/local/server.conf at Install time, with ITS password, which, as you note, will override anything I put into my app directory.

I was going to try one more sequence...

1) Install via msiexec command, with the LAUNCHSPLUNK=0 flag, to prevent Splunk from starting and creating server.conf.

2) DELETE etc/system/local/server.conf.

3) Create, and copy files to, my app directory, including a version of server.conf with MY certs and password.

4) Start Splunk.

But I decided to "Ask Splunk" first, since it sure didn't feel like a "Best Practice".

If the above Install sequence works, is that what is recommended? Is there a cleaner, more elegant approach? This one feels a bit "klooge"y. We are developing an automated process to deploy all our Windows forwarders, and we want it to be as solid as reasonably possible.

Thx again.

mfeeny1

0 Karma

Path Finder

That's what I decided to do as well. At least on linux, I can script this.
Here are the important bits for anyone who stumbles onto this post in future (work in progress):

# Repack Splunk UF installer to include needed files
# (passwd, splunk.secret, splunk-launch.conf...)
cd /tmp

FINALFOLDER="splunk_UF_install"
SOURCEFILES="splunk_UF_source_files"
SPLUNKUF=$(ls splunkforwarder-*.tgz)

# unpack Splunk UF
if [ -f $SPLUNKUF ]; then
  # Prep staging folder
  if [ -d $FINALFOLDER ]; then
      rm -rf $FINALFOLDER
  fi
  mkdir $FINALFOLDER
  cp $SPLUNKUF $FINALFOLDER/
  tar -xf /tmp/$FINALFOLDER/$SPLUNKUF -C /tmp/$FINALFOLDER/
  rm /tmp/$FINALFOLDER/$SPLUNKUF
  echo "[OK] extraction of Splunk UF from $SPLUNKUF successful"
else
   echo "[ERROR] extraction of Splunk UF failed ($SPLUNKUF does not exist)"
   exit 1
fi

# copy over needed files
cp /tmp/$SOURCEFILES/passwd /tmp/$FINALFOLDER/splunkforwarder/etc/
cp /tmp/$SOURCEFILES/splunk.secret /tmp/$FINALFOLDER/splunkforwarder/etc/auth/
cp /tmp/$SOURCEFILES/splunk-launch.conf /tmp/$FINALFOLDER/splunkforwarder/etc/
cp -r /tmp/$SOURCEFILES/my_targetDeploymentServer_app /tmp/$FINALFOLDER/splunkforwarder/etc/apps/
cp -r /tmp/$SOURCEFILES/my_ssl_config_app /tmp/$FINALFOLDER/splunkforwarder/etc/apps/
touch /tmp/$FINALFOLDER/splunkforwarder/etc/splunk.version
echo "[OK] modifications DONE"

# tar the splunkforwarder dir
cd /tmp/$FINALFOLDER && tar -czf splunkforwarder.tgz splunkforwarder && cd /tmp
rm -rf /tmp/$FINALFOLDER/splunkforwarder
echo "[OK] splunkforwarder compressed successfully"

# Add instructions and final tar
cp /tmp/$SOURCEFILES/install_instructions.txt /tmp/$FINALFOLDER/
tar -czf $FINALFOLDER.tgz $FINALFOLDER/splunk_UF_install_instructions.txt $FINALFOLDER/splunkforwarder.tgz
echo "[OK] overall package created successfully"

# Cleaning up
rm -rf /tmp/$FINALFOLDER
rm -rf /tmp/$SOURCEFILES
echo "[OK] Cleanup done. Exiting..."

exit 0
0 Karma

Splunk Employee
Splunk Employee

Do the encoded representations for server.conf/sslKeysfilePassword and outputs.conf/sslPassword match? To verify this, compare the output of "%SPLUNKHOME%\bin\splunk cmd btool server list sslConfig --debug" and "%SPLUNKHOME%\bin\splunk cmd btool outputs list tcpout-server --debug"