Installation

After upgrading to Splunk 6.6, the splunkd.pid file is unreadable. How to fix "Permission denied" errors?

jhl226116
Explorer

Hi Guys,

I upgraded Splunk from 6.5 to 6.6 since then I'm unable to start Splunk properly, it seems there is some issue in the permission level.

Could you walk me through in steps what I suppose to do to fix the permission denied error message pls? I'm just few months into Splunk so fairly new to everything.

See the error message below:

root@indexer:/opt/splunk/bin# ./splunk status

Warning: cannot create "/opt/splunk/var/log/splunk"

Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Labels (2)

sk314
Builder

As root, run chown -R splunk:splunk /opt/splunk
remove any *.pid files from /opt/splunk/var/run/splunk/
if you enabled boot start on Splunk , run service splunk start
if you did not do that, sudo -H -u splunk /opt/splunk/bin/splunk start

0 Karma

jhl226116
Explorer

I have just removed splunkd.pid file from below location and tried to start splunk but still getting permission error.

root@indexer:/opt/splunk/var/run/splunk# ls -l
total 68
drwx--x--- 4 splunk splunk 4096 Mar 30 14:31 appserver
-rw------- 1 splunk splunk 10990 Jun 26 10:54 composite.xml
drwx------ 2 splunk splunk 4096 Mar 30 14:31 csv
drwx--x--x 43 splunk splunk 20480 Jun 23 18:26 dispatch
drwx------ 2 splunk splunk 4096 Jun 22 18:09 merged
drwx------ 4 splunk splunk 4096 Mar 30 14:32 scheduler
-rw------- 1 splunk splunk 556 Jun 23 18:26 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468
-rw------- 1 splunk splunk 0 Jun 23 16:14 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468.lock
-rw------- 1 splunk splunk 15 Jun 26 10:54 splunkd.pid
drwx------ 2 splunk splunk 4096 Jun 22 18:09 srtemp
drwx------ 2 splunk splunk 4096 May 25 18:44 typeahead
drwx------ 2 splunk splunk 4096 Mar 30 14:31 upload

indexer@indexer:~$ sudo -H -u splunk /opt/splunk/bin/splunk start
[sudo] password for indexer:

This command can only be run by bootstart user.

root@indexer:/opt/splunk/bin# ./splunk start

Warning: cannot create "/opt/splunk/var/log/splunk"

Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Be an IT superhero. Go home early.

Checking prerequisites...
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Warning: cannot create "/opt/splunk/var/lib/splunk"

0 Karma

sk314
Builder

did you change the permissions first? the chown command?

0 Karma

jhl226116
Explorer

I've ran the chown command, removed splunkd.pid and tried to start splunk from root.

chown -R splunk:splunk /opt/splunk

0 Karma

sk314
Builder

can you check if the mongod process is running? was it started by the splunk user?

0 Karma

jhl226116
Explorer

how do you check?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

First, lets check what user Splunk is configured to run as..

Look in $SPLUNK_HOME/etc/splunk-launch.conf - you'll see SPLUNK_OS_USER= XXXXX . You need to make sure that this matches the permissions on the file system, and that is who you have to sudo su - XXXXX to for starting splunk.

Check and make sure your splunk user exists in /etc/passwd, ( the user in the above..)

# cat /etc/passwd | grep XXXXX

You should have a user returned with that name, default is splunk.. Set permissions and retry with that user.

0 Karma

jhl226116
Explorer

Initially when I checked in $SPLUNK_HOME/etc/splunk-launch.conf user name was csoc but then I changed it to indexer yesterday since I was having some issues.
It's currently still set as the indexer. SPLUNK_OS_USER=indexer

When I ran cat grep command on all 3 (csoc, indexer, splunk), it only brought results for indexer and splunk.

root@indexer:~# cat /etc/passwd | grep indexer
indexer:x:1000:1000:indexer,,,:/home/indexer:/bin/bash

root@indexer:~# cat /etc/passwd | grep splunk
splunk:x:1001:1001::/home/splunk:

Version 6.5.2

Modify the following line to suit the location of your Splunk install.

If unset, Splunk will use the parent of the directory containing the splunk

CLI executable.

SPLUNK_HOME=/home/build/build-home/ivory

By default, Splunk stores its indexes under SPLUNK_HOME in the

var/lib/splunk subdirectory. This can be overridden

here:

SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk

Splunkd daemon name

SPLUNK_SERVER_NAME=Splunkd

Splunkweb daemon name

SPLUNK_WEB_NAME=splunkweb

If SPLUNK_OS_USER is set, then Splunk service will only start

if the 'splunk [re]start [splunkd]' command is invoked by a user who

is, or can effectively become via setuid(2), $SPLUNK_OS_USER.

(This setting can be specified as username or as UID.)

SPLUNK_OS_USER

SPLUNK_OS_USER=indexer

Ok, where to go from here?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

So from here...

(root)# chown -R indexer:indexer /opt/splunk
(root)# sudo su - indexer
(indexer)# /opt/splunk/bin/splunk start

See what happens. If this fails, Im leaning towards saying youve got a disk volume issue or something else going on in your OS...

jhl226116
Explorer

Awesome! it's working now. I was able to start Splunk with your commands.

root@indexer:/opt/splunk/bin# ./splunk start

Splunk> CSI: Logfiles.

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...

Done

Waiting for web server at http://127.0.0.1:8000 to be available..... Done

If you get stuck, we're here to help.

Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://indexer:8000

root@indexer:/opt/splunk/bin# ./splunk status
splunkd is running (PID: 5391).
splunk helpers are running (PIDs: 5392 5401 5521 5561 9931 9932).

So what does (root)# chown -R indexer:indexer /opt/splunk command do can you explain to me?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

This changes the ownership for both user and group, to indexer, for all folders below /opt/splunk. Since Splunk was trying to start as the indexer user, this user needed permissions for read, write, and execute on all bits under the /opt/splunk folder.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Seems to be a permissions issue with your installation. Most likely you're running Splunk itself as the 'splunk' user. When you upgraded, you ran the upgrade as root, and permissions were effected. Heres a quick way to fix..

(root)# killall splunk && killall mongod
(root)# chown -R splunk:splunk /opt/splunk
(root)# sudo su - splunk
(splunk)# /opt/splunk/bin/splunk start
In the case of this post, the user was indexer

This assumes you're running Splunk as the user 'splunk'. (which is typically default.) Read more on this here :
http://docs.splunk.com/Documentation/Splunk/6.6.1/Installation/RunSplunkasadifferentornon-rootuser

It's a bit odd though that as root you cant start this. Is this a mounted volume? Or local disk?

jhl226116
Explorer

I tried your commands and now stuck in below where it's prompting me for password which don't work with the one that I setup. Is there a default password for this?

root@indexer:~# killall splunk & killall mongod
[1] 27060
splunk: no process found
mongod: no process found
[1]+ Exit 1 killall splunk

root@indexer:~# chown -R splunk:splunk /opt/splunk

root@indexer:~# sudo su - splunk
No directory, logging in with HOME=/

$ /opt/splunk/bin/splunk start
This command can only be run by bootstart user.

$ sudo -i
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
sudo: 3 incorrect password attempts

Splunk instance is running on a vmware workstation.

0 Karma

mbuehler_splunk
Splunk Employee
Splunk Employee

So you need to look into the /etc/init.d/splunk service, and determine who was the original bootstart user, that is the user that the files need to be owned by.

Then the user that you need to use to change the permissions needs to have sudo privileges, so they need to have the ability to make changes to the user permissions. This might be the root user in your case.

0 Karma

jhl226116
Explorer

I think the original user name has been changed since after running esix's command above. I know what my original user name is which was not splunk but something else.

root@indexer:/opt/splunk/etc/init.d# ls -l
total 4
-r--r--r-- 1 splunk splunk 819 May 19 07:37 README

So where to go from here?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...