How do we enable a forwarder boot-start?

[ddrillic]

We are running the following -

/opt/splunk/splunkforwarder/bin/splunk enable boot-start -user splnkfwd

The generated file is -

# cat /etc/init.d/splunk
#!/bin/sh
#
# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
#
# chkconfig: 2345 90 60
# description: Splunk indexer service
#
RETVAL=0

. /etc/init.d/functions

splunk_start() {
  echo Starting Splunk...
  "/opt/splunk/splunkforwarder/bin/splunk" start --no-prompt --answer-yes
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  "/opt/splunk/splunkforwarder/bin/splunk" stop
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  "/opt/splunk/splunkforwarder/bin/splunk" restart
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  "/opt/splunk/splunkforwarder/bin/splunk" status
  RETVAL=$?
}
case "$1" in
  start)
    splunk_start
    ;;
  stop)
    splunk_stop
    ;;
  restart)
    splunk_restart
    ;;
  status)
    splunk_status
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

Where is the binding of splunk to splnkfwd?

One SA suggested to run the following -

/opt/splunk/splunkforwarder/bin/splunk enable boot-start -user splnkfwd
sed -i '/RETVAL/a USER=splnkfwd' /etc/init.d/splunk

What do we need to do?

Running -

/opt/splunk/splunkforwarder/bin/splunk boot-start -user splnkfwd
Command error: The subcommand '-user' is not valid for command 'boot-start'.
Data forwarding configuration management tools.
  Commands:
      enable local-index [-parameter <value>] ...
      disable local-index [-parameter <value>] ...
      display local-index
      add forward-server server
      remove forward-server server
      list forward-server
  Objects:
      forward-server       a Splunk forwarder to forward data to be indexed
      local-index          a local search index on the Splunk server

Matt said that it's all here -

$ cat splunk-launch.conf
#   Version 6.5.1

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
# SPLUNK_HOME=/home/build/build-home/ivory

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
# SPLUNK_DB=/home/build/build-home/ivory/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=SplunkForwarder

# Splunkweb daemon name
SPLUNK_WEB_NAME=splunkweb

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER
SPLUNK_OS_USER=splnkfwd

For one thing, Splunk Enterprise 7.0 System Administration should probably change -

[vagaleraj]

How do we do boot-start in windows system?

[ddrillic]

A side question, besides creating the /etc/init.d/splunk file, does the /opt/splunk/splunkforwarder/bin/splunk enable boot-start -user splnkfwd do anything else?

Because we want to advise our SAs, to simply deploy the modified script @richgalloway provided and bypass running the command.

[ddrillic]

It's interesting.

-- When 'splunk enable boot-start -user ' is invoked, SPLUNK_OS_USER
is set to as a side effect.

It's at splunk-launch.conf.spec

And we speak about it at Why is /opt/splunkforwarder/var/ owned by root on install when nothing else is?

[ddrillic]

An update - we do the following these days -

As root -
$SPLUNK_HOME/bin/splunk enable boot-start -user <splunk user>

In addition, we add USER=<splunk user> to /etc/init.d/splunk underneath the RETVAL=0 entry.

[richgalloway · Accepted solution]

I'm surprised Splunk hasn't fixed how it creates the boot-start file. Try this one:

#!/bin/sh
RETVAL=0
 
. /etc/init.d/functions
 
splunk_start() {
  echo Starting Splunk...
  su - splnkfwd -c '"/opt/splunk/bin/splunk" start --no-prompt --answer-yes'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  su - splnkfwd  -c '"/opt/splunk/bin/splunk" stop'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  su - splnkfwd -c '"/opt/splunk/bin/splunk" restart'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  su - splnkfwd -c '"/opt/splunk/bin/splunk" status'
  RETVAL=$?
}
case "$1" in
start)
   splunk_start
   ;;
stop)
   splunk_stop
   ;;
restart)
   splunk_restart
   ;;
status)
    splunk_status
    ;;
esac
 
exit $RETVAL

[ddrillic]

Interesting @richgalloway - let's keep in mind that all works fine with the out-of-the-box /etc/init.d/splunk file while splunk-launch.conf makes the binding at SPLUNK_OS_USER=splnkfwd.

[richgalloway]

IME, the OOTB file does not always work. I've seen Splunk start under root at boot-time. The above file fixes that. Not that there might not be another way.

[ddrillic]

Very interesting - yesterday we checked a server that had this /etc/init.d/splunk file but the splunk process was not started 15 days ago when the server was rebooted. We ended up rebooting the server a couple of times and in these cases, Splunk did come up. Confusing ; -)