splunk enable boot-start -user in Splunk 6.1.1

[ofwsplunk]

Hi folks..

just did a clean 6.1.1 install.. is it just me? or is "splunk enable boot-start -user " broken?

it used to create a startup script in /etc/init.d/splunk with "/bin/su - " everywhere in the script.. but this time the "su -" parts are all missing and caused me some grief with splunkd not starting...

ie... 6.1.1... /etc/init.d/splunk

splunk_start() {
  echo Starting Splunk...
  "/app/splunk/bin/splunk" start --no-prompt --answer-yes
  RETVAL=$?
}
splunk_stop() {
  echo Stopping Splunk...
  "/app/splunk/bin/splunk" stop
  RETVAL=$?
}
splunk_restart() {
  echo Restarting Splunk...
  "/app/splunk/bin/splunk" restart
  RETVAL=$?
}

version 6.0... /etc/init.d/splunk (where user specified is 'splunk'

splunk_start() {
  echo Starting Splunk...
  /bin/su - splunk -c "\"/app/splunk/bin/splunk\" start --no-prompt --answer-yes"
  RETVAL=$?
}
splunk_stop() {
  echo Stopping Splunk...
  /bin/su - splunk -c "\"/app/splunk/bin/splunk\" stop "
  RETVAL=$?
}
splunk_restart() {
  echo Restarting Splunk...
  /bin/su - splunk -c "\"/app/splunk/bin/splunk\" restart "
  RETVAL=$?
}

[tugnet]

Hi!

I ran into serious trouble with this new method of starting splunk. I do

./splunk enable boot-start -user splunk

and now my running splunkforwarder cannot access the /var/log/messages, secure etc. anymore, even though they are accessible to the group of the splunk user:

-rw-r-----. 1 root splunk    145 Jun 29 03:37 messages

With the old method of /bin/su - splunk -c "\"/opt/splunk/bin/splunk\" start --no-prompt --answer-yes" everything was fine.

any ideas - except setting 644 😉 ?

[Lowell]

I've run into similar issues, though not with the "splunk" group directly. I assigned a "splunklog" group I used for logs, of which, the "splunk" user is a member (along with a few other admins). The problems seems to within the user switching approach within splunk binary, the groups don't get taken into account. Same thing happens with ulimits. If you launch splunk directly (by using sudo/su) then all works as expected; but not so if you launch as root. I too reverted back to the old-style init.d scripts to avoid this issue. Oh, and I've also randomly seen that files created by splunk (indexes, config files, ...) sometimes have the group of root assigned to them. Weird.

[edekker]

I've seen the same thing as both Lowell and johntobin. We recently upgraded from 6.0.2 to 6.1.3, and wound up with a number of files I had to chown to the splunk user.

Additionally, we have run into a permissions issue when it starts up:

WARN FilesystemChangeWatcher - error reading directory "/path/to/syslogs": Permission denied

The splunk user is part of a group which has read only access to these files. Unfortunately, with the new init script setup and the SPLUNK_OS_USER (which is set properly in /opt/splunkforwarder/etc/splunk-launch.conf) this fails to start up.

Workarounds seem to be as previously stated:

1) su to the splunk user and start it with /opt/splunkforwarder/bin/splunk start
OR
2) Revert to the old init script.

Either of these work.

Just wanted to point out this had not been fixed yet as of 6.1.3.

[johntobin]

We've seen the same problem: indexes and config files owned by root. Probably going to have to switch back to the old style startup script to be sure - but this seems like a bug, currently.

[tugnet]

(and SELinux is turned off), OS: CentOS 6.5

[tugnet]

i forgot: /etc/splunk-launch.conf is ok, SPLUNK_OS_USER=splunk is set.

[wethackrey]

We've opened a case for this: 171993
... but as we do more research, this looks like it may be the new expected behavior in 6.1.1.

We note that the enable boot-start -user <> command adds the stanza mentioned by jensmartin to /etc/splunk-launch.conf:

SPLUNK_OS_USER=<<username>>

This causes Splunk 6.1.1 to start as the < user even without an su to that user in /etc/init.d/splunk.

Also, interestingly, if we set the -user flag to a user unknown to the OS, Splunk complains:

[splunk@sharepath bin]$ ./splunk enable boot-start -user elmonteslimm
Overwriting present value (splunk) of SPLUNK_OS_USER in         
/opt/splunkforwarder/etc/splunk-launch.conf
su: user elmonteslimm does not exist
First-time-run has not finished.  Ignore this error when previewing migration - exiting.

...but it adds a SPLUNK_OS_USER stanza for that user to splunk-launch.conf:

SPLUNK_OS_USER=elmonteslimm

Note the following comments are in the splunk-launch.conf file:

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)

[jensmartin]

To solve this I edited:
/opt/splunkforwarder/etc/splunk-launch.conf

And set the SPLUNK_OS_USER option in that file.
I packaged a version of this file in my install script to not have to edit it on all installs.

[wethackrey]

I'm seeing the same thing and came here to Answers to see if anyone else has run into this:

If I run

./splunk enable boot-start -user splunk

on a Scientific Linux machine with Splunk 6.0.3 installed, the command works as expected, creating a splunk script in /etc/init.d that does an su to the splunk user:

splunk_start() {
    echo Starting Splunk...
    /bin/su - splunk -c "\"/opt/splunk/bin/splunk\" start --no-prompt --answer-yes"
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}

If I do the same thing on a different Scientific Linux machine with Splunk 6.1.1 installed, the /etc/init.d/splunk script gets created, but the su - splunk switch to the splunk user is not added. It's as if the enable boot-start command was run without the -user flag.

splunk_start() {
    echo Starting Splunk...
    "/opt/splunk/bin/splunk" start --no-prompt --answer-yes
    RETVAL=$?
    [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}

If I run

chkconfig --list | grep splunk

it returns the same results on both servers:
splunk 0:off 1:off 2:on 3:on 4:on 5:on 6:off
splunk 0:off 1:off 2:on 3:on 4:on 5:on 6:off

I'm running both commands as root from within $SPLUNK_HOME/bin. These are different servers, of course, and the environments are not identical. Both are 64 bit SL:

Splunk 6.0.3: Linux somosomo 2.6.32-220.4.1.el6.x86_64 #1 SMP Mon Jan 23 17:20:44 CST 2012 x86_64 x86_64 x86_64 GNU/Linux

Splunk 6.1.1: Linux SL64X86X64-00 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 20:37:17 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

I'm setting up another server to test with now, but this is feeling like a 6.1 (or 6.1.1) issue.

[jensmartin]

I am seeing the same thing.