Hi all. I am preparing for a production install of Splunk 6.1.1 (new install, 2 IDX, 2 SH w/ SHP, DS/License server VM, and a rSyslog VM with 105GB daily license) and am having an issue in a lab environment, that I have a less than desirable workaround for. I have opened a case with Splunk, but wanted to share my issue here as well to see if anyone has seen this before.
SUBJECT:
Running Splunk as non-root (RHEL 6.x) is not working properly
DESCRIPTION:
I am building out a dev environment of Splunk 6.1.1 to ensure my process is good prior to building out our production Splunk 6.1.1 environment. I am using the Splunk 64-bit Redhat RPM build, and when I tell Splunk to start as the 'splunk' user, it fails to start because it doesn't have permission to write to '$SPLUNK_HOME/var/logs/splunk/first_install.log'. first_install.log is create with 'root' as the owner, so 'splunk' does not have access to write to the file. If I chown that log file to 'splunk:splunk', splunk will now enable boot-start. However, upon rebooting the server, Splunk does not start. If I try to manually start Splunk using '/etc/init.d/splunk start', I get a similar error where permissions are denied when trying to write to splunkd-utility.log. Again, if I 'chown -R splunk;splunk /apps/splunk', then Splunk will successfully start.
Now it's my understanding that in Splunk 6.1, Splunk was changed to where it will start as root, and switch to a named account ('splunk', non-privileged), but it seems these log files are being created with root as owner. This is stopping me from successfully installing Splunk per the install guide, without running chown twice and I fear that this will cause other issues later on.
This issue is holding up our production deployment of Splunk.
STEPS REPRODUCE:
http://pastebin.com/pjijUpyb
In the pastebin you will see the full commands for my install, and my workaround. I am using a named account called 'splunker', which has sudoer rights. In my environment, once a *NIX based system goes production, only the *NIX admins have access to the root account so I have to use a named account with sudoer access.
Anyone run into this, or am I just being over-concerned about this?
According to support, this has been submitted to the devs as a bug. No idea what will come of it, but here is a workaround that is working for me.
sudo rpm -i --prefix=/apps splunk-6.1.1-207789-linux-2.6-x86_64.rpm
sudo -u splunk /apps/splunk/bin/splunk start --accept-license
sudo /apps/splunk/bin/splunk enable boot-start -user splunk
Starting it manually as user splunk for the first time seems to fix the log file ownership problem, then the enable boot-start sets all that up. Seems to work just fine for me.
What version of Splunk are you running? Things changed (in my understanding) in version 6.1 that caused this issue for me.
I wonder why you install it as root and try to run as another user. In our production, it's installed and run as a non-root user without any problem.
According to support, this has been submitted to the devs as a bug. No idea what will come of it, but here is a workaround that is working for me.
sudo rpm -i --prefix=/apps splunk-6.1.1-207789-linux-2.6-x86_64.rpm
sudo -u splunk /apps/splunk/bin/splunk start --accept-license
sudo /apps/splunk/bin/splunk enable boot-start -user splunk
Starting it manually as user splunk for the first time seems to fix the log file ownership problem, then the enable boot-start sets all that up. Seems to work just fine for me.