Getting permission error on deployment server

[kiranpanchavat1]

We are getting the below error while running the commands under bin directory. 

Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

xxxx bin]# ./splunk restart
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable. [FAILED]
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Be an IT superhero. Go home early.

Checking prerequisites...
Checking http port [8000]: not available
ERROR: http port [8000] - port is already bound. Splunk needs to use this port.

I have tried the below steps 

Solved: Splunk will not start and is waiting for config lo... - Splunk Community 

[root@xxxxxx bin]# ./splunk clean locks
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

location : 

/opt/splunk/var/run/splunk

-rw-r-----. 1 root root 48 Nov 2 15:00 splunkd.pid

[richgalloway]

First, it's not considered Best Practice to run Splunk as root.

Second, it seems the instructions in the linked answer were not followed since the splunkd.pid file is still present.  Stop Splunk, remove the file, then try restarting Splunk.

[kiranpanchavat1]

We already tried but that file is not moving and also not deleting. 

 splunk]# ls -lrth
total 28K
drwx------. 2 splunk splunk 6 Sep 19 04:43 upload
drwx--x---. 4 splunk splunk 33 Sep 19 04:43 appserver
drwx------. 2 splunk splunk 6 Sep 19 04:43 csv
drwx------. 4 splunk splunk 49 Sep 19 04:43 scheduler
drwx------. 2 splunk splunk 6 Nov 2 03:01 srtemp
-rw-------. 1 splunk splunk 0 Nov 2 14:15 session-2d93841b6f06596921bc36f6cd5d42a003e8acce.lock
-rw-------. 1 splunk splunk 17 Nov 2 14:27 cachemanager_upload.json
drwx------. 2 splunk splunk 62 Nov 2 14:27 merged
-rw-------. 1 splunk splunk 282 Nov 2 14:32 session-2d93841b6f06596921bc36f6cd5d42a003e8acce
drwx------. 2 splunk splunk 6 Nov 2 15:17 search_telemetry
drwx--x--x. 4 splunk splunk 174 Nov 2 15:18 dispatch
-rw-------. 1 splunk splunk 12K Nov 2 15:20 composite.xml
-rw-r-----. 1 splunk splunk 40 Nov 2 15:21 splunkd.pid.bkp_11_2_2021
-rw-r-----. 1 root root 40 Nov 2 15:26 splunkd.pid
splunk]# cp splunkd.pid.bkp_11_2_2021 splunkd.pid
cp: overwrite 'splunkd.pid'? yes
 splunk]# ls -lrth
total 28K
drwx------. 2 splunk splunk 6 Sep 19 04:43 upload
drwx--x---. 4 splunk splunk 33 Sep 19 04:43 appserver
drwx------. 2 splunk splunk 6 Sep 19 04:43 csv
drwx------. 4 splunk splunk 49 Sep 19 04:43 scheduler
drwx------. 2 splunk splunk 6 Nov 2 03:01 srtemp
-rw-------. 1 splunk splunk 0 Nov 2 14:15 session-2d93841b6f06596921bc36f6cd5d42a003e8acce.lock
-rw-------. 1 splunk splunk 17 Nov 2 14:27 cachemanager_upload.json
drwx------. 2 splunk splunk 62 Nov 2 14:27 merged
-rw-------. 1 splunk splunk 282 Nov 2 14:32 session-2d93841b6f06596921bc36f6cd5d42a003e8acce
drwx------. 2 splunk splunk 6 Nov 2 15:17 search_telemetry
drwx--x--x. 4 splunk splunk 174 Nov 2 15:18 dispatch
-rw-------. 1 splunk splunk 12K Nov 2 15:20 composite.xml
-rw-r-----. 1 splunk splunk 40 Nov 2 15:21 splunkd.pid.bkp_11_2_2021
-rw-r-----. 1 root root 40 Nov 2 15:26 splunkd.pid
 splunk]# mv splunkd.pid /tmp
 splunk]# ls
appserver composite.xml dispatch scheduler session-2d93841b6f06596921bc36f6cd5d42a003e8acce splunkd.pid srtemp
cachemanager_upload.json csv merged search_telemetry session-2d93841b6f06596921bc36f6cd5d42a003e8acce.lock splunkd.pid.bkp_11_2_2021 upload
 splunk]# ls -lrth
total 28K
drwx------. 2 splunk splunk 6 Sep 19 04:43 upload
drwx--x---. 4 splunk splunk 33 Sep 19 04:43 appserver
drwx------. 2 splunk splunk 6 Sep 19 04:43 csv
drwx------. 4 splunk splunk 49 Sep 19 04:43 scheduler
drwx------. 2 splunk splunk 6 Nov 2 03:01 srtemp
-rw-------. 1 splunk splunk 0 Nov 2 14:15 session-2d93841b6f06596921bc36f6cd5d42a003e8acce.lock
-rw-------. 1 splunk splunk 17 Nov 2 14:27 cachemanager_upload.json
drwx------. 2 splunk splunk 62 Nov 2 14:27 merged
-rw-------. 1 splunk splunk 282 Nov 2 14:32 session-2d93841b6f06596921bc36f6cd5d42a003e8acce
drwx------. 2 splunk splunk 6 Nov 2 15:17 search_telemetry
drwx--x--x. 4 splunk splunk 174 Nov 2 15:18 dispatch
-rw-------. 1 splunk splunk 12K Nov 2 15:20 composite.xml
-rw-r-----. 1 splunk splunk 40 Nov 2 15:21 splunkd.pid.bkp_11_2_2021
-rw-r-----. 1 root root 40 Nov 2 15:27 splunkd.pid
 splunk]# cd /opt/splunk/bin
 bin]# ./splunk status
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

[isoutamo]

Hi
probably your FS has gone to read only mode for some reason. Try to check that from dmesg and/or syslog messages. Then fix the reason remount FS again with rw option and try again to clean pid files and then restart splunk.
r. Ismo