How can I create a password file that can be automated to share amongst my heavy forwarders?
Ah, one of my favorite things to do.
$SPLUNK_HOME/etc/system/default/user-seed.conf can be loaded with a default username and password so the normal
admin account is created with a username/password you define in that
user-seed.conf file. You'll see splunk deletes that clear-text file after start up. Therefore, you can hardcode those values in that file at install.
Alternatively, you can create a deployement app that contains a new passwd file and a scripted input to backup the old passwd file and put the new on in place. Scripted input means the output will be collected by splunk (define that sourcetype!). Keep in mind that the scripted input could be configured to run at splunk start OR on a schedule. Either way you'll want to take into account a restart after the file is in place AND removing the app (remove the server from the server class) OR don't remove the app to make sure the password file stays what you want it to be. Of course, for this, you'll also need the splunk.secret file to be distributed just the same so as to ensure the hash in the passwd file is honored.
A different option that might get to your same solution is to have a scripted input that disables the passwd file (by creating an empty one) and have a global app that distributes the ldap auth config. The result is that the only way to log in to the forwarder (or any instance really) is with a valid credential WHICH can therefore be audited (unlike a local account).
Bounce back with questions and good luck!
@212830 - any questions or comments thus far? Looking to get you an acceptable answer.
To deploy secure passwords across multiple servers, please refer to this topic in Splunk docs:
Hope it helps. Thanks!