I've stood up an index cluster and am testing some things at the moment. I used the splunk.secret from the cluster master instance and seeded it to all of the indexers before their installations were started so they all share the same encryption salt. I did this per this article: docs.splunk.com/Documentation/Splunk/6.2.8/Security/Deploysecurepasswordsacrossmultipleservers
I would like to distribute SSL certificates signed by my CA to the cluster peers to enable various SSL communications. Here is what happens when I try to do this:
I created config bundle which includes the rootca.pem file and the concatenated clustercert.pem file with all of the pertinent certs. The bundle also includes a server.conf with the following information:
Notice how I have the password currently unencrypted in the server.conf file, which is not ideal for a number of reasons. Since all of the cluster peers share the same splunk.secret I should be able to put the encrypted password in this server.conf file which would be distributed to the peers. Here is where I get hung up. I don't know if it's possible to manually encrypt a password string with the splunk.secret file. Here's how I've tried to get around this:
Put the above stanzas in the cluster master's server.conf file and restart, but it refuses to encrypt the password (I'm assuming because it's not actually using any cluster replication so it isn't recognizing the need for those stanzas and password?) Otherwise, if it did encrypt the password, I would then use it in the config bundle that would be distributed. (Note: adding these stanzas didn't seem to adversely affect the cluster master, it appeared to flat out ignore them)
Push the config bundle out with the unencrypted password. The cluster peers take the bundle, hash the password, but because /slave-apps/ is Read Only it writes the hashed password in /local/server.conf. I can then take the hashed password from there, replace the unencrypted password in the bundle, delete the stanza from all of the indexers /local/server.conf, and then redeploy the bundle with the correctly hashed password. This DOES work. The cluster peers easily decrypt the hashed password and do not try to write anything else to /local/server.conf. This is NOT a great solution, but it's the only one I've been able to come up with that actually works. Right now I'm only testing this in a cluster with one Cluster Master and two peers but I plan on deploying certs to a 15+ peers in my production environment.
Anyone have any ideas? Is there a way to hash passwords manually using the splunk.secret salt file?
#Jimmy J - 07/15/2008
#Modified this script to use the /bin/bash shell as opposed to the /bin/sh shell. The /bin/sh does not interpret the -n flag
#correctly on mac i.e. the trailing newline is added to the password
#No need of a key file to save the key used for encrypting/decrypting the passwords.
#We now use the splunk.secret key that comes with every installation of splunk
echo -n "Enter the password you want to encrypt:"
echo -n "Enter it one more time to make sure you typed it in correctly:"
if [ $pass = $pass2 ]; then
echo -n $pass | openssl bf -e -a -pass file:$SPLUNK_HOME/etc/auth/splunk.secret
echo "Copy the string on the line above and paste it into"
echo "local/imap.conf as the value for xpassword."
echo "Sorry, your passwords did not match."
Essentially it's using the openssl blowfish cipher to encrypt in base64 using the splunk.secret as the password source. One thing to consider is that the splunk.secret will be different between all of your index cluster members, so there is that to consider.
Hopefully this helps you out. It certainly helped me out.