Root Can't Create /var/log files

[heats]

This is the first time this has come up:

When running the following command as root:

(10:07:49) root@servername:/opt/splunkforwarder/bin
--> ./splunk enable boot-start -user splunk

Warning: cannot create "/opt/splunkforwarder/var/log/splunk"

Warning: cannot create "/opt/splunkforwarder/var/log/introspection"
First-time-run has not finished. Ignore this error when previewing migration - exiting.

Any idea what could be causing this? Root permissions should have what's needed to create the var/log/ files

[jkat54 · Accepted solution]

Apparently someone has set the permissions on /opt/splunkforwarder so that only the owner of the directory can write files/folders under it.

For example:

chmod 700 /opt/splunkforwarder

would make it so that only the owner can read/write/execute it.

You can do the following to get around this but ultimately your permissions need to be fixed.

1st stop Splunk if it's running

/opt/splunkforwarder/bin/splunk stop

2nd, make root the owner of the Splunk dir:

chown -Rf root. /opt/splunkforwarder

3rd, run the same boot start command

/opt/splunkforwarder/bin/splunk enable boot-start -user splunk

4th, change ownership back to splunk user

chown -Rf splunk. /opt/splunkforwarder

5th, switch to Splunk user

su splunk

6th, restart Splunk

/opt/splunkforwarder/bin/splunk start

[joshualemoine]

Absolutely brilliant. I have been searching for this answer for quite some time. Thanks so much. The only extra step I had to do what kill the pid running splunk at the very end before su to splunk and starting splunk, b/c I couldn't stop splunk at the beginning of this sequence of commands due to the "unable to create introspection, var/log/splunk, and this was even as the root user! This all started from an improper clone of a server. Thanks again!

[jkat54]

anytime, thanks for the upvote(s)!

[esix_splunk]

Looks to me like there was a problem with the initial installation, or someone installed the forwarder as a different runtime user, and then restarted it as root.

You can try fixing permissions first, chown -R splunk:splunk /opt/splunkforwarder, then sudo to the splunk user and try running /opt/splunkforwarder/bin/splunk start.

See if you get those errors still. Count to that you can chown that to root:root, and run splunk start as root and see if you get the same errors.