ERROR: kvstore port [8191] - port is already bound. Splunk needs to use this port.

[Hemnaath]

Hi All, I am getting this below Error, when i am trying to restart the splunk services on one of the Heavy forwarder instances , I am getting the below error message.

[splunk@splunk01 bin]$ ./splunk restart
splunkd is not running. [FAILED]

Splunk> All batbelt. No tights.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: not available
ERROR: kvstore port [8191] - port is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]:

When checked whether port is used by any other application by executing the ps -aux | grep 8191 and I found that mongod process taking up the port 8191. Need to know whether i can kill the below mongod process id and restart the splunk services.

[splunk@splunk01 bin]$ ps -aux | grep 8191
splunk 24597 0.4 0.5 1366300 95464 ? Ssl Nov10 71:26 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=EA0E64F9-39BF-4B45-9876-C14227BD1429 --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --sslCipherConfig=TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH --nounixsocket --noscripting
splunk 47877 0.0 0.0 112648 964 pts/0 S+ 09:09 0:00 grep --color=auto 8191

kill -9 24597

Kindly guide me on this.

[indreshdowjones]

do we know what is the permanent solution ?

[Hemnaath · Accepted solution]

Hi Niketnilay, hey thanks for your effort on this, now splunk service is up and running fine in HF instance. I had followed the below steps to restart the splunk service.

steps:

1) First need to check wether the port is used be some other application by executing the ps -aux | grep

ps -aux | grep 8191

[splunk@splunk01 bin]$ ps -aux | grep 8191
splunk 24597 0.4 0.5 1366300 95464 ? Ssl Nov10 71:26 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=EA0E64F9-39BF-4B45-9876-C14227BD1429 --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --sslCipherConfig=TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH --nounixsocket --noscripting
splunk 47877 0.0 0.0 112648 964 pts/0 S+ 09:09 0:00 grep --color=auto 8191

2) Before going for the kill, I had made sure that the splunkd services are not running at the back ground by executing the below command.
cd /opt/splunk/bin
./splunk status
splunkd is not running.

ps -ef | grep -i splunkd
splunk 48308 48267 0 09:42 pts/0 00:00:00 grep --color=auto -i splunkd

3) After executing the above steps, then we have executed Kill 24597 (mongod process id) and before restarting the splunk services by executing the below command. I had made sure that mongod process is properly killed.

ps -ef | grep -i mongod
splunk 48328 48267 0 09:45 pts/0 00:00:00 grep --color=auto -i mongod

4) Started the splunk service using the splunk user id.
cd /opt/splunk/bin
./splunk start

5) Validated the splunk service are up and running or not.

cd /opt/splunk/bin
./splunk status
splunkd is running (PID: 48375).
splunk helpers are running (PIDs: 48381 48407 48527 48547 48569).

[niketn]

@Hemnaath after restart did you check the process running on 8191?

[Hemnaath]

Yes I had checked it by executing the ps -aux | grep 8191 and I could see the mongod is using the process id but parent process id is used by splunk. When checked the other splunk instances, I could see the same result.

ps -ef | grep -i splunk

splunk 51435 51412 0 10:08 ? 00:00:10 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=EA0E64F9-39BF-4B45-9876-C14157BD1429 --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --sslCipherConfig=TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH --nounixsocket --noscripting

ps -aux | grep 51412
splunk 51412 0.0 0.0 71068 8468 ? Ss 10:08 0:00 [splunkd pid=51407] splunkd -p 8089 restart [process-runner]

I hope you are asking to this only.

thanks

[niketn]

Yes. I was. Just wanted to make sure. I feel your answer is so detailed that it should be the one which should be Accepted as answer. So let me convert your comment to answer so that you can accept the same. You can up vote my comment if it helped 🙂

[niketn]

@Hemnaath, KV Store uses mongod, so may be splunk is confused with mongod running as a different process rather than KV Store's.

You can stop splunkd to see whether mongod is removed or not. If not then try to kill it and restart Splunk.

If still Splunk does not start, you can also try to change the port from Splunk > Settings > Server Settings > General Settings> KV Store Port#