Deployment Architecture

Change splunk.secret

FritzWittwer_ol
Contributor

It is not official supported, but we have to try to change secret.conf on a few thousand universal forwarders. A complete deinstallation and a new installation is not an option as this would reread all the log files we are indexing. We must do this as we are no longer allowed to have the clear text passwords of certificates in our apps we are deploying.

The Steps would be:

  • upgrade the universal forwarder from 5.x and 6.x to 6.4.3 (replaces all the binaries)
  • change deploymentclient.conf to point to a new deployment server. This new deployment server will provide the same apps as the old one, but has hashes matching the newly installed splunk.secret instead of the clear text passwords.
  • replyce the splunk.secret (and the password file)
  • restart the universal forwarder.

The follwoing command seem to work:

export SPLUNK_HOME=/opt/splunkforwarder
cp <file with defined secret> $SPLUNK_HOME/etc/auth/splunk.secret
rm ${SPLUNK_HOME}/etc/auth/ca.pem
rm ${SPLUNK_HOME}/etc/auth/ca.srl
rm ${SPLUNK_HOME}/etc/auth/cacert.pem
rm ${SPLUNK_HOME}/etc/auth/server.pem
rm ${SPLUNK_HOME}/etc/system/local/server.conf
touch ${SPLUNK_HOME}/ftr
<update deploymentcleint.conf>
/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports

This removes anything with hashed passwords except the ones in our apps which we replace afterwards, and forces the universal forwarder to recreate them on the start as it behaves like a first start.

Its ugly, but seems to work in a test environment, do we have to expect any gotchas from this?

traxxasbreaker
Communicator

Make sure you also update the sslKeysfilePassword in the sslConfig stanza of the server.conf file in system/local with the default in plain text and let Splunk re-hash it with the new splunk.secret. I think starting with 6.5 the setting changes to sslPassword instead but I found that the upgrade did that conversion. That lets Splunk properly decrypt the passwords for the default certs it ships with that it uses for communications on the management port, which may affect use of the forwarder's REST API and its ability to communicate with the deployment server.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...