This is the first time this has come up:
When running the following command as root:
(10:07:49) root@servername:/opt/splunkforwarder/bin
--> ./splunk enable boot-start -user splunk
Warning: cannot create "/opt/splunkforwarder/var/log/splunk"
Warning: cannot create "/opt/splunkforwarder/var/log/introspection"
First-time-run has not finished. Ignore this error when previewing migration - exiting.
Any idea what could be causing this? Root permissions should have what's needed to create the var/log/ files
Apparently someone has set the permissions on /opt/splunkforwarder so that only the owner of the directory can write files/folders under it.
For example:
chmod 700 /opt/splunkforwarder
would make it so that only the owner can read/write/execute it.
You can do the following to get around this but ultimately your permissions need to be fixed.
1st stop Splunk if it's running
/opt/splunkforwarder/bin/splunk stop
2nd, make root the owner of the Splunk dir:
chown -Rf root. /opt/splunkforwarder
3rd, run the same boot start command
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk
4th, change ownership back to splunk user
chown -Rf splunk. /opt/splunkforwarder
5th, switch to Splunk user
su splunk
6th, restart Splunk
/opt/splunkforwarder/bin/splunk start
Apparently someone has set the permissions on /opt/splunkforwarder so that only the owner of the directory can write files/folders under it.
For example:
chmod 700 /opt/splunkforwarder
would make it so that only the owner can read/write/execute it.
You can do the following to get around this but ultimately your permissions need to be fixed.
1st stop Splunk if it's running
/opt/splunkforwarder/bin/splunk stop
2nd, make root the owner of the Splunk dir:
chown -Rf root. /opt/splunkforwarder
3rd, run the same boot start command
/opt/splunkforwarder/bin/splunk enable boot-start -user splunk
4th, change ownership back to splunk user
chown -Rf splunk. /opt/splunkforwarder
5th, switch to Splunk user
su splunk
6th, restart Splunk
/opt/splunkforwarder/bin/splunk start
Absolutely brilliant. I have been searching for this answer for quite some time. Thanks so much. The only extra step I had to do what kill the pid running splunk at the very end before su to splunk and starting splunk, b/c I couldn't stop splunk at the beginning of this sequence of commands due to the "unable to create introspection, var/log/splunk, and this was even as the root user! This all started from an improper clone of a server. Thanks again!
anytime, thanks for the upvote(s)!
Looks to me like there was a problem with the initial installation, or someone installed the forwarder as a different runtime user, and then restarted it as root.
You can try fixing permissions first, chown -R splunk:splunk /opt/splunkforwarder, then sudo to the splunk user and try running /opt/splunkforwarder/bin/splunk start.
See if you get those errors still. Count to that you can chown that to root:root, and run splunk start as root and see if you get the same errors.