Deployment Architecture

Why do I need to unbind the mgmt port in order to restart splunk?

splunktrainingu
Communicator

I was under the impression that port 8089 is used to manage the apps on your endpoints using the Settings > Forwarder Management. This is what happens when I tried to restart splunk forwarder 

 

./splunk restart
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunkforwarder/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Like an F-18, bro.

Checking prerequisites...
Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
Checking mgmt port [8089]: Cannot initialize: /opt/splunkforwarder/etc/apps/learned/metadata/local.meta: Permission denied
not available
ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]: n
Exiting....

 

 

I am currently testing with a one of the Linux servers, I have my "deploymentclient.conf" file in splunkforwarder/etc/system/local/ and it is set to port 8089.  My main server is a single deployment on prem. I am not sure what I am doing wrong? 

 

I tried to mimic the set up of my windows servers because they have a "deploymentclient.conf" file in their splunkforwarder/etc/system/local directory.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Most probably your splunk was running with root/high privileged user and you are trying to restart the instance with another user (with less privileges).

Check the file ownership and make sure that it's the user you are using and not root(or any other)

 

Happy Splunking!

splunktrainingu
Communicator

That is not the case, I tried with my username and then I tried with the owner username.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Ok, can you check these

1. Do a file listing for below and check who owns the file ?

/opt/splunkforwarder/var/run/splunk/splunkd.pid

 2. Do a ps -eaf |grep splunkd and check owner of the process

While restarting splunk, make sure that you are using the user from the above.

Also try a splunk stop and check the output

Happy Splunking!
0 Karma

splunktrainingu
Communicator

Splunkd.pid did not exist in the directory 

/opt/splunkforwarder/var/run/splunk/

I got this when I ran the command

ps -eaf |grep splunkd
splunk   18031 17753  0 12:29 pts/4    00:00:00 grep splunkd

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...