Why are we experiencing this Splunk Permission Error

[jbender72]

Hello,

Noticed my Indexer was down and I could not sign in.  Went to restart splunk as sudo then root user and got this error:

Splunk is unable to write to the directory /opt/splunk and therefore will not run. Please check for appropriate permissions on this directory and its contents as necessary.

Checked what I could, I cannot come up with a solution.  Did not find much on the Internet.  Please help.

[Ocelot]

I came across this error today. It turned out to be a disk space issue. Provisioning more storage allowed Splunk to start without issue. 

[Jumy]

I am having same issue, how do I provision for the disk space issue? Thanks

chown: changing ownership of '/opt/splunk/.bash_history': Read-only file system
chown: changing ownership of '/opt/splunk': Read-only file system

[nickhills]

I am pleased you have it working, but for the record, chmod'ding to 777 is not a sensible fix for a production system. 🙂

[nickhills]

Thats odd.

Can you run "sudo id splunk"

It should return something like

uid=xxx(splunk) gid=xxx(splunk) group=xxx(splunk)

 or does it report
"splunk" no such user

[jbender72]

It is giving no such user

[nickhills]

How was splunk installed? rpm/deb or from the tar.gz?

Who owns files in /opt/splunk?

[jbender72]

I tried to set the azureuser as owner of the files who owns splunk.  I chmod 777 and it appeared to work.

[azureuser@cb-spl-in1-p splunk]$ ls -al
total 3200
drwxrwxrwx. 10 azureuser azureuser 237 Dec 15 16:48 .
drwxrwxrwx. 3 root root 20 Nov 13 20:09 ..
drwxrwxrwx. 4 azureuser azureuser 4096 Nov 12 15:37 bin
-rwxrwxrwx. 1 azureuser azureuser 57 Nov 12 15:37 copyright.txt
drwxrwxrwx. 16 azureuser azureuser 4096 Dec 15 15:34 etc
drwxrwxrwx. 4 azureuser azureuser 62 Nov 12 15:37 include
drwxrwxrwx. 8 azureuser azureuser 4096 Nov 12 15:37 lib
-rwxrwxrwx. 1 azureuser azureuser 85709 Nov 12 15:37 license-eula.txt
drwxrwxrwx. 3 azureuser azureuser 58 Nov 12 15:37 openssl
-rwxrwxrwx. 1 azureuser azureuser 844 Nov 12 15:37 README-splunk.txt
drwxrwxrwx. 4 azureuser azureuser 108 Nov 12 15:37 share
-rwxrwxrwx. 1 azureuser azureuser 3168712 Nov 12 15:37 splunk-8.1.0-f57c09e87251-linux-2.6-x86_64-manifest
drwxrwxrwx. 2 azureuser azureuser 54 Nov 12 15:37 swidtag
drwxrwxrwx. 6 azureuser azureuser 52 Nov 12 15:37 var
[azureuser@cb-spl-in1-p splunk]$

[jbender72]

as tar

[nickhills]

In that case, unless you set up the splunk user by hand, it wont exist.

At a guess you were therefore running Splunk as root.
If root has permission issues then you probably have bigger problems!

Can you access the contents of /opt/splunk?

[jbender72]

Yes I can access the contents, just no splunk user anymore?

[nickhills]

Was Splunk configured to run as "root"?

If you configured Splunk to run as "splunk" (recommended) you should not start it as root. If you do, it may mess about with the folder permissions, which means the next time you start it as "splunk" you get permission errors.

If this is what happened, you should stop Splunk (if running) then (assuming linux)

sudo chown -R splunk:splunk /opt/splunk

Then start Splunk as "splunk" - probably easier to reboot the system and let the Splunk boot-start process handle it.

[jbender72]

I don't know if this is in the correct area.  I am getting this: invalid user.

azureuser@cb-spl-in1-p ~]$ sudo chown -R splunk:splunk /opt/splunk
chown: invalid user: ‘splunk:splunk’

Am I placing the command in the incorrect place?