Change splunk.secret

[FritzWittwer_ol]

It is not official supported, but we have to try to change secret.conf on a few thousand universal forwarders. A complete deinstallation and a new installation is not an option as this would reread all the log files we are indexing. We must do this as we are no longer allowed to have the clear text passwords of certificates in our apps we are deploying.

The Steps would be:

• upgrade the universal forwarder from 5.x and 6.x to 6.4.3 (replaces all the binaries)
• change deploymentclient.conf to point to a new deployment server. This new deployment server will provide the same apps as the old one, but has hashes matching the newly installed splunk.secret instead of the clear text passwords.
• replyce the splunk.secret (and the password file)
• restart the universal forwarder.

The follwoing command seem to work:

export SPLUNK_HOME=/opt/splunkforwarder
cp <file with defined secret> $SPLUNK_HOME/etc/auth/splunk.secret
rm ${SPLUNK_HOME}/etc/auth/ca.pem
rm ${SPLUNK_HOME}/etc/auth/ca.srl
rm ${SPLUNK_HOME}/etc/auth/cacert.pem
rm ${SPLUNK_HOME}/etc/auth/server.pem
rm ${SPLUNK_HOME}/etc/system/local/server.conf
touch ${SPLUNK_HOME}/ftr
<update deploymentcleint.conf>
/opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports

This removes anything with hashed passwords except the ones in our apps which we replace afterwards, and forces the universal forwarder to recreate them on the start as it behaves like a first start.

Its ugly, but seems to work in a test environment, do we have to expect any gotchas from this?

[traxxasbreaker]

Make sure you also update the sslKeysfilePassword in the sslConfig stanza of the server.conf file in system/local with the default in plain text and let Splunk re-hash it with the new splunk.secret. I think starting with 6.5 the setting changes to sslPassword instead but I found that the upgrade did that conversion. That lets Splunk properly decrypt the passwords for the default certs it ships with that it uses for communications on the management port, which may affect use of the forwarder's REST API and its ability to communicate with the deployment server.