Deployment of SSL Certificates on Splunk Universal Forwarders

[michaeltay]

Hi Splunkers!

I would like to secure splunkd (port 8089) on Splunk Universal Forwarders by using a throwaway self-signed certificate.

I tried the following methods:
1) Using msiexec to install Splunk Universal Forwarder, and also include the throwaway certificate for the forwarders

msiexec.exe /i splunkforwarder-<version>.msi DEPLOYMENT_SERVER="<deployment_server>:8089" AGREETOLICENSE=Yes CERTFILE=<throwaway forwarder certificate>.pem CERTPASSWORD=<private key password> /quiet

This method will install Splunk Universal Forwarder, and add the certificate into $SPLUNK_HOME\etc\auth. However, after installation, it still uses the default Splunk certificate in $SPLUNK_HOME\etc\system\local\server.conf.
2) Deploy an app containing server.conf to the deployment clients

[sslConfig]
serverCert = $SPLUNK_HOME\etc\apps\ssl_app\cert\<throwaway forwarder certificate>.pem
sslPassword = <private key password>
sslVersions = tls

I understand this method does not work, as the configuration in $SPLUNK_HOME\etc\system\local\server.conf will replace any configuration done in the app.

May I know the following:
a) What is the best way to configure Splunk Universal Forwarders to use a self-signed certificate for splunkd during installation?
b) What is the best way to configure Splunk Universal Forwarders to use a self-signed certificate for splunkd after installation?

Thanks!

[p_gurav]

This may help you :

https://docs.splunk.com/Documentation/Splunk/7.0.3/Security/ConfigureSplunkforwardingtousesignedcert...

[pmalcakdoj]

If you name the files exactly like Splunk does, it will work.
If your source files are in C:\temp for example, use:

msiexec.exe /i splunkforwarder... CERTFILE=C:\temp\server.pem ROOTCACERTFILE=C:\temp\cacert.pem ... /quiet

If they are called server.pem and cacert.pem respectively, they will overwrite the default splunk-generated ones.

Passwords is where this gets interesting. You have couple options:

• provide installer with CERTPASSWORD=super_secret_pw server.pem needs to have its priv key encrypted with super_secret_pw
• do not provide installer with CERTPASSWORD flag at all server.pem will need to have its priv key encrypted with "password" (because reasons)
• don't provide installer with CERTPASSWORD and swap the encrypted sslPassword in local\server.conf for your cleartext password (it will be encrypted on next restart)

My preferred method is to give installer CERTFILE=C:\temp\server.pem with encrypted priv key and omitting CERTPASSWORD entirely. You'll want to stop splunk from launching with LAUNCHSPLUNK=0 so that system\local\server.conf isn't generated yet. This gives you opportunity to replace splunk.secret with your own (known) version and copy/paste your encrypted sslPassword. This way, neither your priv key nor cleartext password is ever revealed to whoever runs the installer script.

To answer your second question:
If splunk finds server.conf in one of the apps BEFORE fist launch (hence the importance of LAUNCHSPLUNK=0), it will NOT create system\local\server.conf. I took advantage of this and created few apps that get copy/pasted alongside the install. You'll have to be careful with naming them because of settings precedence, but crafted correctly, you can create your own defaults that live in apps and have system/local completely empty.
Name your defaults app zzzSystemLocalReplacement. Leave this app unmanaged by DS. Because it starts with "zzz" it will be matched as a last resort. Then, you can use deployment server to push down different app with new SSL certs when the time comes. New SSL cert app will then take higher precedence and become your effective configuration.
This procedure has few key advantages:

• system\local is totally empty
• you have a way to provide your forwarders a "default" config
• DS has ability to push new apps and trump these defaults WITHOUT overwriting them (so you can always fall back on your defaults config if you forget a setting or typo something)