Deployment Architecture

Why are we experiencing this Splunk Permission Error

jbender72
Path Finder

Hello,

 

Noticed my Indexer was down and I could not sign in.  Went to restart splunk as sudo then root user and got this error:

Splunk is unable to write to the directory /opt/splunk and therefore will not run. Please check for appropriate permissions on this directory and its contents as necessary.

Checked what I could, I cannot come up with a solution.  Did not find much on the Internet.  Please help.

Labels (2)
0 Karma

Ocelot
Splunk Employee
Splunk Employee

I came across this error today. It turned out to be a disk space issue. Provisioning more storage allowed Splunk to start without issue. 

Tags (1)
0 Karma

Jumy
New Member

I am having same issue, how do I provision for the disk space issue? Thanks


chown: changing ownership of '/opt/splunk/.bash_history': Read-only file system
chown: changing ownership of '/opt/splunk': Read-only file system

Tags (1)
0 Karma

nickhills
Ultra Champion

I am pleased you have it working, but for the record, chmod'ding to 777 is not a sensible fix for a production system. 🙂

 

If my comment helps, please give it a thumbs up!
0 Karma

nickhills
Ultra Champion

Thats odd.

Can you run "sudo id splunk"

It should return something like

uid=xxx(splunk) gid=xxx(splunk) group=xxx(splunk)


 or does it report
"splunk" no such user

If my comment helps, please give it a thumbs up!

jbender72
Path Finder

It is giving no such user

0 Karma

nickhills
Ultra Champion

How was splunk installed? rpm/deb or from the tar.gz?

Who owns files in /opt/splunk?

 

If my comment helps, please give it a thumbs up!
0 Karma

jbender72
Path Finder

I tried to set the azureuser as owner of the files who owns splunk.  I chmod 777 and it appeared to work.

 

[azureuser@cb-spl-in1-p splunk]$ ls -al
total 3200
drwxrwxrwx. 10 azureuser azureuser 237 Dec 15 16:48 .
drwxrwxrwx. 3 root root 20 Nov 13 20:09 ..
drwxrwxrwx. 4 azureuser azureuser 4096 Nov 12 15:37 bin
-rwxrwxrwx. 1 azureuser azureuser 57 Nov 12 15:37 copyright.txt
drwxrwxrwx. 16 azureuser azureuser 4096 Dec 15 15:34 etc
drwxrwxrwx. 4 azureuser azureuser 62 Nov 12 15:37 include
drwxrwxrwx. 8 azureuser azureuser 4096 Nov 12 15:37 lib
-rwxrwxrwx. 1 azureuser azureuser 85709 Nov 12 15:37 license-eula.txt
drwxrwxrwx. 3 azureuser azureuser 58 Nov 12 15:37 openssl
-rwxrwxrwx. 1 azureuser azureuser 844 Nov 12 15:37 README-splunk.txt
drwxrwxrwx. 4 azureuser azureuser 108 Nov 12 15:37 share
-rwxrwxrwx. 1 azureuser azureuser 3168712 Nov 12 15:37 splunk-8.1.0-f57c09e87251-linux-2.6-x86_64-manifest
drwxrwxrwx. 2 azureuser azureuser 54 Nov 12 15:37 swidtag
drwxrwxrwx. 6 azureuser azureuser 52 Nov 12 15:37 var
[azureuser@cb-spl-in1-p splunk]$

0 Karma

jbender72
Path Finder

as tar

0 Karma

nickhills
Ultra Champion

In that case, unless you set up the splunk user by hand, it wont exist.

At a guess you were therefore running Splunk as root.
If root has permission issues then you probably have bigger problems!

Can you access the contents of /opt/splunk?

If my comment helps, please give it a thumbs up!
0 Karma

jbender72
Path Finder

Yes I can access the contents, just no splunk user anymore?

0 Karma

nickhills
Ultra Champion

Was Splunk configured to run as "root"?

If you configured Splunk to run as "splunk" (recommended) you should not start it as root. If you do, it may mess about with the folder permissions, which means the next time you start it as "splunk" you get permission errors.

If this is what happened, you should stop Splunk (if running) then (assuming linux)

sudo chown -R splunk:splunk /opt/splunk

 

Then start Splunk as "splunk" - probably easier to reboot the system and let the Splunk boot-start process handle it.

 

If my comment helps, please give it a thumbs up!

jbender72
Path Finder

I don't know if this is in the correct area.  I am getting this: invalid user.

azureuser@cb-spl-in1-p ~]$ sudo chown -R splunk:splunk /opt/splunk
chown: invalid user: ‘splunk:splunk’

Am I placing the command in the incorrect place?

 

0 Karma