We are currently creating a Disaster Recovery guide for our Splunk installation that must contain all steps necessary to fail over splunk in our environment. One task is to roll all hot buckets to warm. Obviously we need to be "logged in" on the CLI before we can execute this step. The problem is that we need to use a username and password that exists in splunk to get this done and we would have to add that username and password to the DR guide as well to make sure in case of a disaster anyone with basic system knowledge would be able to execute the steps.
What is the best way to do this?
Can we disable the password functionality on the CLI?
should we put the admin users credentials into the DR guide?
No you can not disable this functionality. It's up to you where you document admin passwords required for DR, but I would imagine that Splunk passwords are not the only ones needed. Another thing you could do it provide steps to reset the default admin password by replacing the etc/passwd file.