Splunk hashes the sslPassword making it so that the key file can't be read

[Charlie]

Along the lines of this question - http://splunk-base.splunk.com/answers/11577/splunk-overwrites-outputsconf-and-inputsconf-on-reboot, but I've tried a few additional things I'd like to note.

To reiterate the issue - I am trying to enable SSL in my outputs.conf for one of my forwarders. I am using the default certs and the default password "password".

I've changed the location of my outputs.conf quite a few times, trying these paths:

1) ./etc/system/local/outputs.conf

2) ./etc/apps/forwarder/local/outputs.conf

3) ./etc/apps/forwarder/default/outputs.conf

No matter which of these paths I choose, I continue to run into this pattern:

1) Update outputs.conf to have a sslPassword of "password"

2) Use btool to check outputs and note the sslPassword:

./bin/splunk cmd btool outputs list --debug
forwarder  [tcpout]
system     autoLB = true
forwarder  defaultGroup = splunk
system     forwardedindex.0.whitelist = .*
system     forwardedindex.1.blacklist = _.*
system     forwardedindex.2.whitelist = _audit
system     forwardedindex.filter.disable = false
system     maxQueueSize = 500KB
forwarder  [tcpout-server://server.domain:8002]
forwarder  sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
forwarder  sslPassword = password
forwarder  sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
forwarder  [tcpout:splunk]
forwarder  compressed = true
forwarder  disabled = false
forwarder  server = server.domain:8002

3) Restart splunkd - ./bin/splunk restart splunkd

4) Use btool to check outputs again and note the sslPassword again:

./bin/splunk cmd btool outputs list --debug
forwarder  [tcpout]
system     autoLB = true
forwarder  defaultGroup = splunk
system     forwardedindex.0.whitelist = .*
system     forwardedindex.1.blacklist = _.*
system     forwardedindex.2.whitelist = _audit
system     forwardedindex.filter.disable = false
system     maxQueueSize = 500KB
forwarder  [tcpout-server://server.domain:8002]
forwarder  sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
forwarder  sslPassword = $1$ZMTWcdnuueG6
forwarder  sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
forwarder  [tcpout:splunk]
forwarder  compressed = true
forwarder  disabled = false
forwarder  server = server.domain:8002

The sslPassword has been hashed, just as it mentions here - http://www.splunk.com/wiki/Community:Splunk2Splunk_SSL_DefaultCerts in the note for #2.

Note that the server certificate pass
phrase will be hashed and stored in
$SPLUNK_HOME/etc/system/local/outputs.conf,
overwriting the clear-text value of
"sslPassword" if it was defined there.
If "sslPassword" was defined in
clear-text in an outputs.conf located
in an app, it will not be hashed
there and will still be present in
clear text in that location. This
doesn't matter too much in this case
since the pass phrase for the default
server certificate is well known.

The note claims that the password will not be hashed if located in an app. But it is hashed when I used any of the locations above, where I consider paths 2 and 3 to be an app path.

I don't have an issue with the hashing, but I feel that it has to do with the SSL error I am getting:

ERROR SSLCommon - Can't read key file
$SPLUNK_HOME/etc/auth/server.pem
errno=101077092 error:06065064:digital
envelope
routines:EVP_DecryptFinal_ex:bad
decrypt.

When I check the password of the server.pem file using openssl:

openssl rsa -in /logs/splunk_forwarder/etc/auth/server.pem -text
Enter pass phrase for /logs/splunk_forwarder/etc/auth/server.pem:
I enter "password"

It works.

So either, the hashing needs to stop or needs to work.

[jrstear · Accepted solution]

Doing splunk add forward-server <host:port> -ssl-cert-path /path/ssl.crt -ssl-root-ca-path /path/ca.crt -ssl-password <password>
as described at http://docs.splunk.com/Documentation/Splunk/4.2.4/Deploy/Deployanixdfmanually works. Then you can move etc/local/{outputs,server}.conf (which contain the hashed password) to app dirs if desired, and restart.

[yannK]

This is because the default password is in the "system/default/"
And at start splunk encrypts it and save to "system/local/"

Now the seed used for the encryption can be different on every instance, therefore the encrypted password is different. A solution to avoid it is to uniformize the seed when installing ($SPLUNK_HOME\etc\auth\splunk.secret), or use the previous method.

[Charlie]

Surprisingly - using the CLI vs modifying the files directly does work. Thank you.