I installed Splunk using a domain account. It was a successful installation, I was able to install 2 add-ons and deploy few forwarders to get data in.
I have logged into Splunk and clicked "Restart Splunk" because I did not see data from one server. "Restart Splunk" timed out and failed. Now I am not able to start the application.
Splunkd and splunkweb services won't start using the domain account that was used in the initial setup (I am able to start it using Local System account). The failure code I am getting is "Error 1069" which indicates that the credentials are incorrect. Nothing has been changed before restart and I double checked the credentials. I checked if the account is not locked and even changed the credential in AD to make sure they are correct.
I have used Splunk recommended way of starting the application through the command line. It goes through without any errors but the splunkd service does not start.
This happened to me when I was forced to change the password of the machine. Apparently, the password change wasn't propagated to the splunkforwarder service. I went to Start, Administrative Tools, Services, and selected the SplunkForwarder service and double-clicked to open it. Then, on the logon tab, I updated the password to the new one that I had changed it to. After that, I was able to start the splunkforwarder by typing ".\splunk start" from the c:\Program Files\SplunkUniversalForwarder\bin directory.
The symptom I was getting on the Splunk side was that it would attempt to start the forwarder, and everything would pass correctly and it would look like it was going to start, but then it would just say "SplunkForwarder: Stopped". It wasn't until I looked into the Windows logs to see the error 1069: The service did not start due to a logon failure.
Could you double-check the rights of that user on the local system?
While there are a few places this could go wrong, my first thought given your description is that there is a group policy in place restricting certain local rights to certain accounts. This is unlikely to disallow setting the account rights properly initially, but at GP refresh, every hour and a half by default, the rights get restricted again. I believe this doesn't affect running processes, but will affect that process/user the next time the process starts.
And that sounds a lot like what you may have experienced.
Permission to log on as a service.
Permission to log on as a batch job.
Permission to replace a process-level token.
Permission to act as part of the operating system.
Permission to bypass traverse checking.
If that works, please mark this as answered so that others can benefit from this answer as well. Thanks!