Hi Guys,
I upgraded Splunk from 6.5 to 6.6 since then I'm unable to start Splunk properly, it seems there is some issue in the permission level.
Could you walk me through in steps what I suppose to do to fix the permission denied error message pls? I'm just few months into Splunk so fairly new to everything.
See the error message below:
root@indexer:/opt/splunk/bin# ./splunk status
Warning: cannot create "/opt/splunk/var/log/splunk"
Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
splunkd.pid file is unreadable.
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
As root, run chown -R splunk:splunk /opt/splunk
remove any *.pid files from /opt/splunk/var/run/splunk/
if you enabled boot start on Splunk , run service splunk start
if you did not do that, sudo -H -u splunk /opt/splunk/bin/splunk start
I have just removed splunkd.pid file from below location and tried to start splunk but still getting permission error.
root@indexer:/opt/splunk/var/run/splunk# ls -l
total 68
drwx--x--- 4 splunk splunk 4096 Mar 30 14:31 appserver
-rw------- 1 splunk splunk 10990 Jun 26 10:54 composite.xml
drwx------ 2 splunk splunk 4096 Mar 30 14:31 csv
drwx--x--x 43 splunk splunk 20480 Jun 23 18:26 dispatch
drwx------ 2 splunk splunk 4096 Jun 22 18:09 merged
drwx------ 4 splunk splunk 4096 Mar 30 14:32 scheduler
-rw------- 1 splunk splunk 556 Jun 23 18:26 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468
-rw------- 1 splunk splunk 0 Jun 23 16:14 session-86d3a80d3dbec0935375ba9adeb589bfc7ef3468.lock
-rw------- 1 splunk splunk 15 Jun 26 10:54 splunkd.pid
drwx------ 2 splunk splunk 4096 Jun 22 18:09 srtemp
drwx------ 2 splunk splunk 4096 May 25 18:44 typeahead
drwx------ 2 splunk splunk 4096 Mar 30 14:31 upload
indexer@indexer:~$ sudo -H -u splunk /opt/splunk/bin/splunk start
[sudo] password for indexer:
This command can only be run by bootstart user.
root@indexer:/opt/splunk/bin# ./splunk start
Warning: cannot create "/opt/splunk/var/log/splunk"
Warning: cannot create "/opt/splunk/var/log/introspection"
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Splunk> Be an IT superhero. Go home early.
Checking prerequisites...
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Cannot initialize: /opt/splunk/etc/apps/Splunk_CiscoSecuritySuite/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/splunk_instrumentation/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Checking configuration... Done.
Creating: /opt/splunk/var/lib/splunk
Warning: cannot create "/opt/splunk/var/lib/splunk"
did you change the permissions first? the chown command?
I've ran the chown command, removed splunkd.pid and tried to start splunk from root.
chown -R splunk:splunk /opt/splunk
can you check if the mongod process is running? was it started by the splunk user?
how do you check?
First, lets check what user Splunk is configured to run as..
Look in $SPLUNK_HOME/etc/splunk-launch.conf - you'll see SPLUNK_OS_USER= XXXXX . You need to make sure that this matches the permissions on the file system, and that is who you have to sudo su - XXXXX to for starting splunk.
Check and make sure your splunk user exists in /etc/passwd, ( the user in the above..)
# cat /etc/passwd | grep XXXXX
You should have a user returned with that name, default is splunk.. Set permissions and retry with that user.
Initially when I checked in $SPLUNK_HOME/etc/splunk-launch.conf user name was csoc but then I changed it to indexer yesterday since I was having some issues.
It's currently still set as the indexer. SPLUNK_OS_USER=indexer
When I ran cat grep command on all 3 (csoc, indexer, splunk), it only brought results for indexer and splunk.
root@indexer:~# cat /etc/passwd | grep indexer
indexer:x:1000:1000:indexer,,,:/home/indexer:/bin/bash
root@indexer:~# cat /etc/passwd | grep splunk
splunk:x:1001:1001::/home/splunk:
SPLUNK_SERVER_NAME=Splunkd
SPLUNK_WEB_NAME=splunkweb
SPLUNK_OS_USER=indexer
Ok, where to go from here?
So from here...
(root)# chown -R indexer:indexer /opt/splunk
(root)# sudo su - indexer
(indexer)# /opt/splunk/bin/splunk start
See what happens. If this fails, Im leaning towards saying youve got a disk volume issue or something else going on in your OS...
Awesome! it's working now. I was able to start Splunk with your commands.
root@indexer:/opt/splunk/bin# ./splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-6.6.1-aeae3fe0c5af-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
Waiting for web server at http://127.0.0.1:8000 to be available..... Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://indexer:8000
root@indexer:/opt/splunk/bin# ./splunk status
splunkd is running (PID: 5391).
splunk helpers are running (PIDs: 5392 5401 5521 5561 9931 9932).
So what does (root)# chown -R indexer:indexer /opt/splunk command do can you explain to me?
This changes the ownership for both user and group, to indexer, for all folders below /opt/splunk. Since Splunk was trying to start as the indexer user, this user needed permissions for read, write, and execute on all bits under the /opt/splunk folder.
Seems to be a permissions issue with your installation. Most likely you're running Splunk itself as the 'splunk' user. When you upgraded, you ran the upgrade as root, and permissions were effected. Heres a quick way to fix..
(root)# killall splunk && killall mongod
(root)# chown -R splunk:splunk /opt/splunk
(root)# sudo su - splunk
(splunk)# /opt/splunk/bin/splunk start
In the case of this post, the user was indexer
This assumes you're running Splunk as the user 'splunk'. (which is typically default.) Read more on this here :
http://docs.splunk.com/Documentation/Splunk/6.6.1/Installation/RunSplunkasadifferentornon-rootuser
It's a bit odd though that as root you cant start this. Is this a mounted volume? Or local disk?
I tried your commands and now stuck in below where it's prompting me for password which don't work with the one that I setup. Is there a default password for this?
root@indexer:~# killall splunk & killall mongod
[1] 27060
splunk: no process found
mongod: no process found
[1]+ Exit 1 killall splunk
root@indexer:~# chown -R splunk:splunk /opt/splunk
root@indexer:~# sudo su - splunk
No directory, logging in with HOME=/
$ /opt/splunk/bin/splunk start
This command can only be run by bootstart user.
$ sudo -i
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
Sorry, try again.
[sudo] password for splunk:
sudo: 3 incorrect password attempts
Splunk instance is running on a vmware workstation.
So you need to look into the /etc/init.d/splunk service, and determine who was the original bootstart user, that is the user that the files need to be owned by.
Then the user that you need to use to change the permissions needs to have sudo privileges, so they need to have the ability to make changes to the user permissions. This might be the root user in your case.
I think the original user name has been changed since after running esix's command above. I know what my original user name is which was not splunk but something else.
root@indexer:/opt/splunk/etc/init.d# ls -l
total 4
-r--r--r-- 1 splunk splunk 819 May 19 07:37 README
So where to go from here?