Removing stale pid file... Can't unlink pid file "/opt/splunk/var/run/splunk/splunkd.pid": Permission denied

[vnguyen46]

Hello,
I think there is a permission issue after my Linux system admin changed access to a directory.
Now my SH stopped. Either I run splunk status or start, I received the error message:
Removing stale pid file... Can't unlink pid file "/opt/splunk/var/run/splunk/splunkd.pid": Permission denied
My splunkd.pid file permission set as following:
-rwxrwxrwx. 1 splunk splunk 364 Apr 21 10:50 splunkd.pid

Anyone has resolved the same issue?

Thank you,

[PavelP · Accepted solution]

Hello @vnguyen46,

as @renjith.nair mentioned, you have to check permission of folders.

Here are the permissions and owners of the pid file and folders of default install:

# namei -mo /opt/splunk/var/run/splunk/splunkd.pid
f: /opt/splunk/var/run/splunk/splunkd.pid
 dr-xr-xr-x root   root   /
 drwxr-xr-x root   root   opt
 drwx------ splunk splunk splunk
 drwx--x--- splunk splunk var
 drwx--x--- splunk splunk run
 drwx--x--x splunk splunk splunk
 -rw-r----- splunk splunk splunkd.pid

The permission set rwxrwxrwx (777 in octal notation) is applied usually to links and suid files only and should not be used.

[vnguyen46]

Thank you all. The issue was that the Linux admin renamed file splunkd.pid and assume this file would be recreated once Splunk restarted, but it didn't. Rename the file back to its original doesn't work, so this file needs to be manually recreated.

Thanks,

[renjith_nair]

@vnguyen46 ,
Make sure that the splunk user has permissions on the directory level as well.