Restarting Splunk when Splunk is running as regular user

[agodoy]

Is there a trick to be able to restart Splunk from the web interface when it is not running as root. Are there extra permissions that need to be set that are not covered here.

[dmr195]

If you've run Splunk as two different users and ended up with index files owned by two different users then you can fix this pretty easily from a shell prompt. Suppose Splunk is installed in /opt/splunk and at various times you've run Splunk as OS users root and splunk. Then:

cd /opt/splunk
find . -user root
find . -user splunk

At least one of the find commands will find masses of files, but one of them should find no files. If both find commands find something then you've got mixed ownership of the installation. To fix this, shut down Splunk and then, as root, run:

chown -R splunk /opt/splunk

Then all the files will be consistently owned by the splunk user. Then su to be the splunk user and restart Splunk so it's running as the splunk user. Always run the Splunk processes as the same OS user.

Since Splunk can execute Python scripts, it's probably also best that the OS user that Splunk runs as is not root, because then an accidental or malicious problem in a Python script could trash your machine.

[agodoy]

So it seems like the issue is specific to splunkweb.

Timed out waiting for splunkweb to top. [FAILED]

That is when doing a restart via CLI with the splunk account.

What would be causing this? Oh, BTW, root did not own any of the files in $SPLUNK_HOME.

[bmacias84]

When starting up as a regular user use the the following ./splunk start --debug this will ouput debug on your terminal. if you started splunk as another user the designated daemon/service user you probabily broke permissions.

[agodoy]

Yes, I have restarted splunk as root via CLI. Any idea what logs I can look at to see what file might be causing the issue?

[bmacias84]

I've used those instructions. Have or did you start splunk under a different user context? Its possible that file were created as that user and your regular user doesnt have permission to read/write those files.