Knowledge Management

Why the KVstore process is being started as a root?

abhi04
Path Finder

Splunk is not restarting because we are getting the error "kvstore port [8191] - port is already bound". After I check, I observed the process is starting as a root and so while restarting it assumes the port is being taken by another process. I killed the process and was able to start the splunk.

But I wanted to know the reason and the resolution to prevent this from happening in the future. I have checked and verified that the /var/lib/splunk/kvstore/mongo is owned by splunk. But some of the files such as "admin.0" "admin.ns" "config.0" and "config.ns" are owned as root and not splunk. Wanted to know what are those files and if these permissions should also be changed to splunk.
Also, the splunk.key have proper permission.

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Stop Splunk completely and verify all processes are down "ps -ef |grep -i splunk" e.g.
If any are still active, kill them off.

Modify the config at /opt/splunk/etc/splunk-launch.conf and ensure that SPLUNK_OS_USER is set to splunk.
SPLUNK_OS_USER=splunk

If you are using systemd, also verify the user is set correctly within the unit file in the [Service] stanza
User=splunk

Start Splunk back up and verify.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Did this help resolve your issue? If so, please "accept" the answer so that others in the community may benefit.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

nickhills
Ultra Champion

This can happen if your instance was at some point started by root (perhaps by mistake)
All files in $SPLUNK_HOME should be owned by the user Splunk is running as (splunk)

If you have files inside $SPLUNK_HOME owned by root, you should probably run:
sudo chown -R splunk:splunk /opt/splunk - or the path of $SPLUNK_HOME

If my comment helps, please give it a thumbs up!
0 Karma

abhi04
Path Finder

Hi @nickhillscpl,

The /opt/splunk is already owned as splunk.

I just wanted to know if there is a permanent fix for this. will the re-installation of splunk resolve this permanently?

0 Karma

nnimbe1
Path Finder

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance

0 Karma
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...