With my current Splunk environment running on Linux, what happens with service account password changes?

[mendesjo]

Maybe this is more of a Linux question, however, I have the following scenario and question. We run Splunk on RHEL, and we configured Splunk to start up and run with a non-root account that is also tied into Active Directory (AD). So, when Splunk starts it starts up with this service account and this works fine. When we SSH into one of our Splunk servers we also use this same service account and this works fine also (I would prefer to use individual accounts). My question is around password changes and it's effects to this Splunk service account. Noting the configuration above, when you configure Splunk to start up with this service account, I don't remember every specifying a password in Linux for this account that resides in Active Directory, so when it starts up does it authenticate to AD? We have this running on many servers, and I'm trying to understand the implications of changing this password. Since this same ID is running Splunk on many servers what happens if we were to change the password in AD? Would Splunk stop running? Again, this maybe more of a Linux question. Thanks in advance.

[dwaddle · Accepted solution]

Yes, this is a Unix question and not a Splunk one. Fundamentally, how Linux daemons and Windows services work is similar but there are some rather large differences. One being that, unlike Windows Services, a Linux daemon does not "log in" when the daemon starts. The root user on Linux is able to "become" the service account user without any login password required.

When you did this:

splunk enable boot-start -user someuser

Then you set up Linux to initially start Splunk as root, and the for Splunk itself to use root's ability to become another user to become someuser. Whether someuser is an Active Directory account or not, or what its password is, is not material to the ability for Splunk to run as a process under this user's authority on the operating system.

You can change the password, even delete the account ... and the running process will stay running. (Granted, if you delete the account, then restarting the process later will be much more difficult)

[mendesjo]

Thanks! That really helps. The only thing I don't understand is that if I did delete the account from Active Directory why would Splunk stop running after a restart if the Linux server doesn't authenticate to AD? Also, if it doesn't authenticate I see no reason as to why the password can't change. The only notable exception would be we use LDAP authentication for our users to log on to our search heads and the account is this same service account and if that ID/PASSWORD configured must be good or else when it binds to AD it won't work (besides that condition), do you see a whole in my perception of how this works in theory?

[dwaddle]

why would Splunk stop running after a restart if the Linux server doesn't authenticate to AD?

Unix has a "setuid" system call that allows you to change the user authority a process is running under - this is how Splunk started as root "becomes" the desired user. (It is also the basis for the su and sudo commands.) But, to allow a process to setuid to a user .. that user must exist! This is not an authentication thing, but a user enumeration thing.

if it doesn't authenticate I see no reason as to why the password can't change.

Correct. You can change the AD account password repeatedly and it does not affect Splunk's ability to be started by the OS, because the start process does not "log in".

the account is this same service account and if that ID/PASSWORD configured must be good or else when it binds to AD it won't work

Again, correct. If you do change the AD account's password and Splunk attempts to use that as the LDAP bind account, a password is required there.

[mendesjo]

Thank you so much for taking time to answer my question!