Why do I need to unbind the mgmt port in order to restart splunk?

[splunktrainingu]

I was under the impression that port 8089 is used to manage the apps on your endpoints using the Settings > Forwarder Management. This is what happens when I tried to restart splunk forwarder 

./splunk restart
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Like an F-18, bro.

Checking prerequisites...
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Checking mgmt port [8089]: Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
not available
ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]: n
Exiting....

I am currently testing with a one of the Linux servers, I have my "deploymentclient.conf" file in splunkforwarder/etc/system/local/ and it is set to port 8089.  My main server is a single deployment on prem. I am not sure what I am doing wrong? 

I tried to mimic the set up of my windows servers because they have a "deploymentclient.conf" file in their splunkforwarder/etc/system/local directory.

[renjith_nair]

Most probably your splunk was running with root/high privileged user and you are trying to restart the instance with another user (with less privileges).

Check the file ownership and make sure that it's the user you are using and not root(or any other)

[splunktrainingu]

That is not the case, I tried with my username and then I tried with the owner username.

[renjith_nair]

Ok, can you check these

1. Do a file listing for below and check who owns the file ?

/opt/splunkforwarder/var/run/splunk/splunkd.pid

 2. Do a ps -eaf |grep splunkd and check owner of the process

While restarting splunk, make sure that you are using the user from the above.

Also try a splunk stop and check the output

[splunktrainingu]

Splunkd.pid did not exist in the directory 

/opt/splunkforwarder/var/run/splunk/

I got this when I ran the command

ps -eaf |grep splunkd
splunk   18031 17753  0 12:29 pts/4    00:00:00 grep splunkd