I want to update the splunk.secret on my existing Splunk servers. Gotchas?

[the_wolverine]

For management purposes, I would like the splunk.secret on all my servers to match but its after I have started up Splunk for the first time already. Can I do this without breaking my setup?

The documentation referenced only mentions setting the splunk.secret BEFORE FIRST START. What about after first start?

1) Is there a documented process on how to do this?
2) What should I be aware of?
3) Anyone else done this?

[the_wolverine]

It doesn't appear that Splunk wants to provide the solution for syncing splunk.secret after first start. I'm starting my investigation and will post my findings here. Related question:

http://answers.splunk.com/answers/123896/need-a-list-of-all-the-locations-of-hashed-password-based-o...

[phoenixdigital]

Well from first hand experience it breaks a Search Head.

DO NOT TRY THIS AT HOME!!!

We moved /opt/splunk/etc/passwd out of the way and Splunk recreated that on restart.

Cleartexted any passwords found in conf files with the grep command here
https://answers.splunk.com/answers/123896/need-a-list-of-all-the-locations-of-hashed-password-based-...

However the Search head keeps returning a 500 error.

[piebob · Accepted solution]

the procedure for distributing secure passwords is documented here:
http://docs.splunk.com/Documentation/Splunk/6.0.1/Security/Deploysecurepasswordsacrossmultipleserver...

a good resource for "what gotchas are there?" info in general is this community-authored topic on the Community Wiki: http://wiki.splunk.com/Things_I_wish_I_knew_then

here's a bullet item from that topic about splunk.secret:

"Thinking about search head pooling or clustering? The splunk.secret file is important, because it helps set the encryption key used for things like SSL key files, LDAP service accounts, and so on. For systems that will need to share identical copies of files containing splunk encrypted password data, you may want to copy splunk.secret to such a system before the first time you start Splunk on it."

[the_wolverine]

I have an existing implementation where I want to match up splunk.secret. I have modified my question to reflect that.

"Thinking about search head pooling or clustering? The splunk.secret file is important, because it helps set the encryption key used for things like SSL key files, LDAP service accounts, and so on. For systems that will need to share identical copies of files containing splunk encrypted password data, you may want to copy splunk.secret to such a system before the first time you start Splunk on it."

[woodcock]

@the_wolverine How did this turn out? What did you learn?

[dshpritz]

Also be aware that the splunk.secret is used when storing the credentials for some inputs (like modular inputs).