We have a Splunk service account but we're only supposed to be able to su Splunk. It's not supposed to allow actual sign-ons. I've done password on the splunk account but when attempting ./splunk add forward-server servername.domainname:9997 it keeps prompting me for a Splunk username and credentials.
Is it best practice to assign this account credentials? Does anyone know how to bypass the credential prompting?
When you need authentication to run a splunk command from the CLI, the credentials you will need are for actual Splunk users, not a service account. Typically, you will need a Splunk admin account to add data or, in this case, a forwarder.
You can bypass the credential prompting by passing the credentials through the command line:
./splunk add forward-server servername.domainname:9997 -auth admin:changeme
It's actually prompting for credentials to an account defined in the Splunk instance, not the service account on the OS that the Splunk process is running as. If it's a back-end component, typically it would be the credentials for the admin user.
The question is, are you trying to avoid the prompts to avoid anyone having to know an admin password for this Splunk instance, or to facilitate automation? To avoid people knowing that Splunk admin password you can setup LDAP auth within that Splunk instance and map your Splunk administrator group to the admin role, then each person can use their own credentials to authenticate to the Splunk instance. That approach should also show who did what in the audit logs which is good for accountability anyway, but you would still want at least someone to know that local admin password for the times when it's really needed, like if you need it to fix LDAP auth. Alternatively, if you're looking to avoid the prompts to facilitate automation, you could do something like have the OS account's .bashrc check the credentials out of some password vaulting solution and use them to authenticate to Splunk with the Splunk-level account so anyone who has used
su splunk will automatically be authenticated to the Splunk instance.