topic Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors? in Installation

After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 16 Jun 2020 18:21:50 GMT

Hi Guys,

I upgraded Splunk from 6.5 to 6.6 since then I'm unable to start Splunk properly, it seems there is some issue in the permission level.

Could you walk me through in steps what I suppose to do to fix the permission denied error message pls? I'm just few months into Splunk so fairly new to everything.

See the error message below:

root@indexer:/opt/splunk/bin# ./splunk status

Warning: cannot create "/opt/splunk/var/log/splunk"

Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

esix_splunk — Mon, 26 Jun 2017 04:59:13 GMT

Seems to be a permissions issue with your installation. Most likely you're running Splunk itself as the 'splunk' user. When you upgraded, you ran the upgrade as root, and permissions were effected. Heres a quick way to fix..

(root)# killall splunk && killall mongod
(root)# chown -R splunk:splunk /opt/splunk
(root)# sudo su - splunk
(splunk)# /opt/splunk/bin/splunk start
In the case of this post, the user was indexer

This assumes you're running Splunk as the user 'splunk'. (which is typically default.) Read more on this here :
http://docs.splunk.com/Documentation/Splunk/6.6.1/Installation/RunSplunkasadifferentornon-rootuser

It's a bit odd though that as root you cant start this. Is this a mounted volume? Or local disk?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Mon, 26 Jun 2017 21:12:07 GMT

I tried your commands and now stuck in below where it's prompting me for password which don't work with the one that I setup. Is there a default password for this?

root@indexer:~# killall splunk & killall mongod
[1] 27060
splunk: no process found
mongod: no process found
[1]+ Exit 1 killall splunk

root@indexer:~# chown -R splunk:splunk /opt/splunk

root@indexer:~# sudo su - splunk
No directory, logging in with HOME=/

$ /opt/splunk/bin/splunk start
This command can only be run by bootstart user.

$ sudo -i
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
sudo: 3 incorrect password attempts

Splunk instance is running on a vmware workstation.

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

mbuehler_splunk — Mon, 26 Jun 2017 21:29:40 GMT

So you need to look into the /etc/init.d/splunk service, and determine who was the original bootstart user, that is the user that the files need to be owned by.

Then the user that you need to use to change the permissions needs to have sudo privileges, so they need to have the ability to make changes to the user permissions. This might be the root user in your case.

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Mon, 26 Jun 2017 22:56:59 GMT

I think the original user name has been changed since after running esix's command above. I know what my original user name is which was not splunk but something else.

root@indexer:/opt/splunk/etc/init.d# ls -l
total 4
-r--r--r-- 1 splunk splunk 819 May 19 07:37 README

So where to go from here?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

sk314 — Tue, 27 Jun 2017 01:45:55 GMT

As root, run chown -R splunk:splunk /opt/splunk
remove any *.pid files from /opt/splunk/var/run/splunk/
if you enabled boot start on Splunk , run service splunk start
if you did not do that, sudo -H -u splunk /opt/splunk/bin/splunk start

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 29 Sep 2020 14:39:26 GMT

I have just removed splunkd.pid file from below location and tried to start splunk but still getting permission error.

root@indexer:/opt/splunk/var/run/splunk# ls -l
total 68
drwx--x--- 4 splunk splunk 4096 Mar 30 14:31 appserver
-rw------- 1 splunk splunk 10990 Jun 26 10:54 composite.xml
drwx------ 2 splunk splunk 4096 Mar 30 14:31 csv
drwx--x--x 43 splunk splunk 20480 Jun 23 18:26 dispatch
drwx------ 2 splunk splunk 4096 Jun 22 18:09 merged
drwx------ 4 splunk splunk 4096 Mar 30 14:32 scheduler
-rw------- 1 splunk splunk 556 Jun 23 18:26 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468
-rw------- 1 splunk splunk 0 Jun 23 16:14 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468.lock
-rw------- 1 splunk splunk 15 Jun 26 10:54 splunkd.pid
drwx------ 2 splunk splunk 4096 Jun 22 18:09 srtemp
drwx------ 2 splunk splunk 4096 May 25 18:44 typeahead
drwx------ 2 splunk splunk 4096 Mar 30 14:31 upload

indexer@indexer:~$ sudo -H -u splunk /opt/splunk/bin/splunk start
[sudo] password for indexer:

This command can only be run by bootstart user.

root@indexer:/opt/splunk/bin# ./splunk start

Warning: cannot create "/opt/splunk/var/log/splunk"

Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Be an IT superhero. Go home early.

Checking prerequisites...
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Warning: cannot create "/opt/splunk/var/lib/splunk"

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

sk314 — Tue, 27 Jun 2017 03:50:00 GMT

did you change the permissions first? the chown command?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 27 Jun 2017 03:56:43 GMT

I've ran the chown command, removed splunkd.pid and tried to start splunk from root.

chown -R splunk:splunk /opt/splunk

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

sk314 — Tue, 27 Jun 2017 04:02:37 GMT

can you check if the mongod process is running? was it started by the splunk user?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 27 Jun 2017 04:18:18 GMT

how do you check?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

esix_splunk — Tue, 29 Sep 2020 14:35:32 GMT

First, lets check what user Splunk is configured to run as..

Look in $SPLUNK_HOME/etc/splunk-launch.conf - you'll see SPLUNK_OS_USER= XXXXX . You need to make sure that this matches the permissions on the file system, and that is who you have to sudo su - XXXXX to for starting splunk.

Check and make sure your splunk user exists in /etc/passwd, ( the user in the above..)

# cat /etc/passwd | grep XXXXX

You should have a user returned with that name, default is splunk.. Set permissions and retry with that user.

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 29 Sep 2020 14:35:35 GMT

Initially when I checked in $SPLUNK_HOME/etc/splunk-launch.conf user name was csoc but then I changed it to indexer yesterday since I was having some issues.
It's currently still set as the indexer. SPLUNK_OS_USER=indexer

When I ran cat grep command on all 3 (csoc, indexer, splunk), it only brought results for indexer and splunk.

root@indexer:~# cat /etc/passwd | grep indexer
indexer:x:1000:1000:indexer,,,:/home/indexer:/bin/bash

root@indexer:~# cat /etc/passwd | grep splunk
splunk:x:1001:1001::/home/splunk:

Version 6.5.2

Modify the following line to suit the location of your Splunk install.

If unset, Splunk will use the parent of the directory containing the splunk

CLI executable.

SPLUNK_HOME=/home/build/build-home/ivory

By default, Splunk stores its indexes under SPLUNK_HOME in the

var/lib/splunk subdirectory. This can be overridden

here:

SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk

Splunkd daemon name

SPLUNK_SERVER_NAME=Splunkd

Splunkweb daemon name

SPLUNK_WEB_NAME=splunkweb

If SPLUNK_OS_USER is set, then Splunk service will only start

if the 'splunk [re]start [splunkd]' command is invoked by a user who

is, or can effectively become via setuid(2), $SPLUNK_OS_USER.

(This setting can be specified as username or as UID.)

SPLUNK_OS_USER

SPLUNK_OS_USER=indexer

Ok, where to go from here?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

esix_splunk — Tue, 27 Jun 2017 04:58:19 GMT

So from here...

(root)# chown -R indexer:indexer /opt/splunk
(root)# sudo su - indexer
(indexer)# /opt/splunk/bin/splunk start

See what happens. If this fails, Im leaning towards saying youve got a disk volume issue or something else going on in your OS...

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116 — Tue, 29 Sep 2020 14:40:43 GMT

Awesome! it's working now. I was able to start Splunk with your commands.

root@indexer:/opt/splunk/bin# ./splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done

Waiting for web server at http://127.0.0.1:8000 to be available..... Done

If you get stuck, we're here to help.

Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://indexer:8000

root@indexer:/opt/splunk/bin# ./splunk status
splunkd is running (PID: 5391).
splunk helpers are running (PIDs: 5392 5401 5521 5561 9931 9932).

So what does (root)# chown -R indexer:indexer /opt/splunk command do can you explain to me?

Re: After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

esix_splunk — Thu, 29 Jun 2017 06:33:55 GMT

This changes the ownership for both user and group, to indexer, for all folders below /opt/splunk. Since Splunk was trying to start as the indexer user, this user needed permissions for read, write, and execute on all bits under the /opt/splunk folder.