Getting Data In

Deployment of SSL Certificates on Splunk Universal Forwarders

Path Finder

Hi Splunkers!

I would like to secure splunkd (port 8089) on Splunk Universal Forwarders by using a throwaway self-signed certificate.

I tried the following methods:
1) Using msiexec to install Splunk Universal Forwarder, and also include the throwaway certificate for the forwarders

msiexec.exe /i splunkforwarder-<version>.msi DEPLOYMENT_SERVER="<deployment_server>:8089" AGREETOLICENSE=Yes CERTFILE=<throwaway forwarder certificate>.pem CERTPASSWORD=<private key password> /quiet

This method will install Splunk Universal Forwarder, and add the certificate into $SPLUNK_HOME\etc\auth. However, after installation, it still uses the default Splunk certificate in $SPLUNK_HOME\etc\system\local\server.conf.
2) Deploy an app containing server.conf to the deployment clients

serverCert = $SPLUNK_HOME\etc\apps\ssl_app\cert\<throwaway forwarder certificate>.pem
sslPassword = <private key password>
sslVersions = tls

I understand this method does not work, as the configuration in $SPLUNK_HOME\etc\system\local\server.conf will replace any configuration done in the app.

May I know the following:
a) What is the best way to configure Splunk Universal Forwarders to use a self-signed certificate for splunkd during installation?
b) What is the best way to configure Splunk Universal Forwarders to use a self-signed certificate for splunkd after installation?


0 Karma

Path Finder

If you name the files exactly like Splunk does, it will work.
If your source files are in C:\temp for example, use:

msiexec.exe /i splunkforwarder... CERTFILE=C:\temp\server.pem ROOTCACERTFILE=C:\temp\cacert.pem ... /quiet

If they are called server.pem and cacert.pem respectively, they will overwrite the default splunk-generated ones.

Passwords is where this gets interesting. You have couple options:

  • provide installer with CERTPASSWORD=super_secret_pw server.pem needs to have its priv key encrypted with super_secret_pw
  • do not provide installer with CERTPASSWORD flag at all server.pem will need to have its priv key encrypted with "password" (because reasons)
  • don't provide installer with CERTPASSWORD and swap the encrypted sslPassword in local\server.conf for your cleartext password (it will be encrypted on next restart)

My preferred method is to give installer CERTFILE=C:\temp\server.pem with encrypted priv key and omitting CERTPASSWORD entirely. You'll want to stop splunk from launching with LAUNCHSPLUNK=0 so that system\local\server.conf isn't generated yet. This gives you opportunity to replace splunk.secret with your own (known) version and copy/paste your encrypted sslPassword. This way, neither your priv key nor cleartext password is ever revealed to whoever runs the installer script.

To answer your second question:
If splunk finds server.conf in one of the apps BEFORE fist launch (hence the importance of LAUNCHSPLUNK=0), it will NOT create system\local\server.conf. I took advantage of this and created few apps that get copy/pasted alongside the install. You'll have to be careful with naming them because of settings precedence, but crafted correctly, you can create your own defaults that live in apps and have system/local completely empty.
Name your defaults app zzzSystemLocalReplacement. Leave this app unmanaged by DS. Because it starts with "zzz" it will be matched as a last resort. Then, you can use deployment server to push down different app with new SSL certs when the time comes. New SSL cert app will then take higher precedence and become your effective configuration.
This procedure has few key advantages:

  • system\local is totally empty
  • you have a way to provide your forwarders a "default" config
  • DS has ability to push new apps and trump these defaults WITHOUT overwriting them (so you can always fall back on your defaults config if you forget a setting or typo something)
0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Spotlight on Security: Adoption Motions, War Stories, and More

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Cloud | Empowering Splunk Administrators with Admin Config Service (ACS)

Greetings, Splunk Cloud Admins and Splunk enthusiasts! The Admin Configuration Service (ACS) team is excited ...

Tech Talk | One Log to Rule Them All

One log to rule them all: how you can centralize your troubleshooting with Splunk logs We know how important ...