This is the first time this has come up:
When running the following command as root:
--> ./splunk enable boot-start -user splunk
Warning: cannot create "/opt/splunkforwarder/var/log/splunk"
Warning: cannot create "/opt/splunkforwarder/var/log/introspection"
First-time-run has not finished. Ignore this error when previewing migration - exiting.
Any idea what could be causing this? Root permissions should have what's needed to create the var/log/ files
Looks to me like there was a problem with the initial installation, or someone installed the forwarder as a different runtime user, and then restarted it as root.
You can try fixing permissions first, chown -R splunk:splunk /opt/splunkforwarder, then sudo to the splunk user and try running /opt/splunkforwarder/bin/splunk start.
See if you get those errors still. Count to that you can chown that to root:root, and run splunk start as root and see if you get the same errors.
Apparently someone has set the permissions on /opt/splunkforwarder so that only the owner of the directory can write files/folders under it.
chmod 700 /opt/splunkforwarder
would make it so that only the owner can read/write/execute it.
You can do the following to get around this but ultimately your permissions need to be fixed.
1st stop Splunk if it's running
2nd, make root the owner of the Splunk dir:
chown -Rf root. /opt/splunkforwarder
3rd, run the same boot start command
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk
4th, change ownership back to splunk user
chown -Rf splunk. /opt/splunkforwarder
5th, switch to Splunk user
6th, restart Splunk
Absolutely brilliant. I have been searching for this answer for quite some time. Thanks so much. The only extra step I had to do what kill the pid running splunk at the very end before su to splunk and starting splunk, b/c I couldn't stop splunk at the beginning of this sequence of commands due to the "unable to create introspection, var/log/splunk, and this was even as the root user! This all started from an improper clone of a server. Thanks again!