Why the KVstore process is being started as a root?

[abhi04]

Splunk is not restarting because we are getting the error "kvstore port [8191] - port is already bound". After I check, I observed the process is starting as a root and so while restarting it assumes the port is being taken by another process. I killed the process and was able to start the splunk.

But I wanted to know the reason and the resolution to prevent this from happening in the future. I have checked and verified that the /var/lib/splunk/kvstore/mongo is owned by splunk. But some of the files such as "admin.0" "admin.ns" "config.0" and "config.ns" are owned as root and not splunk. Wanted to know what are those files and if these permissions should also be changed to splunk.
Also, the splunk.key have proper permission.

[codebuilder]

Stop Splunk completely and verify all processes are down "ps -ef |grep -i splunk" e.g.
If any are still active, kill them off.

Modify the config at /opt/splunk/etc/splunk-launch.conf and ensure that SPLUNK_OS_USER is set to splunk.
SPLUNK_OS_USER=splunk

If you are using systemd, also verify the user is set correctly within the unit file in the [Service] stanza
User=splunk

Start Splunk back up and verify.

[codebuilder]

Did this help resolve your issue? If so, please "accept" the answer so that others in the community may benefit.

[nickhills]

This can happen if your instance was at some point started by root (perhaps by mistake)
All files in $SPLUNK_HOME should be owned by the user Splunk is running as (splunk)

If you have files inside $SPLUNK_HOME owned by root, you should probably run:
sudo chown -R splunk:splunk /opt/splunk - or the path of $SPLUNK_HOME

[abhi04]

Hi @nickhillscpl,

The /opt/splunk is already owned as splunk.

I just wanted to know if there is a permanent fix for this. will the re-installation of splunk resolve this permanently?

[nnimbe1]

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance