Is the default behavior for the Splunk Universal Forwarder on Solaris to prompt for credentials on startup?

[johnglass]

We are preparing to roll out the Universal Forwarder to a pilot group of 50 Solaris servers before deploying to the entire 6000 server environment. During testing of our installation script, we're running into a problem where Splunk is prompting for credentials when trying to start the service:

Starting splunk server daemon (splunkd)...
Done

Init script installed at /etc/init.d/splunk.
Init script is configured to run at boot.
Splunk username:

We can manually enter the default Splunk credentials (admin/changeme) and it will start, however this defeats the purpose of an automated rollout. We've installed as root but this may change in the actual deployment. The script runs through the installation steps in the following order:

/usr/splunkforwarder/bin/splunk start --accept-license --no-prompt --answer-yes *
*/usr/splunkforwarder/bin/splunk enable boot-start
/usr/splunkforwarder/bin/splunk set deploy-poll splunk-ds.xyz.net:8089

Can anyone tell me if this is the default behavior of the UF on Solaris, or if we've done something wrong with the install?

[chanfoli]

Many commands will prompt for splunk credentials if splunk is running but won't if it is not. What I have found is that after accept-license is done, you can stop splunk with $SPLUNK_HOME/bin/splunk stop and then apply these config changes without getting prompted, then start splunk with $SPLUNK_HOME/bin/splunk start. Another options is to specify auth parameters on the command line with --auth admin:password but this forces you to code your credentials into your scripts.

[johnglass]

The prompt for credentials appears upon the final start. Are you saying that after accepting the license we should stop splunk then enable boot-start and deploy-poll, restart Splunk and we shouldn't expect the prompt any longer?

If we go down the path of adding the credentials to the script, would specifying the user/password work when starting the server? I just haven't come across that in the documentation. Would it simply be splunk start -auth admin:changeme?

[chanfoli]

Normally you would not get prompted for creds on the actual initial start. After you accept the license splunkd starts and then any of the commands you are executing are requiring a session as long as splunkd is running. If I recall correctly you have to complete the first run and either accept the license manually or start splunk with the options you are using as the first command (start --accept-license --no-prompt --answer-yes ) so that the initial config is set, then stop splunk to reconfigure with the other 2 commands without requiring a session. This has been my experience, you start splunk the first time, then stop it, then reconfigure with enable boot-start and set deploy-poll options then start splunk.

You should not need auth options or be prompted to simply execute splunk start. Or to execute reconfiguration commands with SPLUNK_HOME/bin/splunk if splunkd has been stopped.

Does that make sense? I don't think this is specific to a solaris install, as it sounds like the same behavior I expect in a Linux environment.

BTW, this looks like it should all work as I describe if you are running splunk as root. If you plan to run splunk as a different user (e.g. splunk) then you will need to specify a user on the enable-boot, and I have found that I have also needed to correct ownership issues after running commands similar to your script as root.