Splunk Fails to Start on Boot (init.d)

[jscraig2006]

Greetings,

I am setting up a new 8.2.2 environment, Red Hat 8.1 and trying to get Splunk to start on boot and to run under a different user than root. I can start it up manually under the "splunk" user without any problems but on boot, it does not.

What I have done so far:

$SPLUNK_HOME/bin/splunk enable boot-start -user splunk

in /etc/init.d/splunk

#!/bin/sh
RETVAL=0
 
. /etc/init.d/functions
 
splunk_start() {
  echo Starting Splunk...
  su - splunk -c '"/opt/splunk/bin/splunk" start --no-prompt --answer-yes'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  su - splunk  -c '"/opt/splunk/bin/splunk" stop'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  su - splunk -c '"/opt/splunk/bin/splunk" restart'
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  su - splunk -c '"/opt/splunk/bin/splunk" status'
  RETVAL=$?
}
case "$1" in
start)
   splunk_start
   ;;
stop)
   splunk_stop
   ;;
restart)
   splunk_restart
   ;;
status)
    splunk_status
    ;;
esac
 
exit $RETVAL

in /opt/splunk/etc/splunk-launcher.conf

#   Version 8.2.2

# Modify the following line to suit the location of your Splunk install.
# If unset, Splunk will use the parent of the directory containing the splunk
# CLI executable.
#
SPLUNK_HOME=/opt/splunk

# By default, Splunk stores its indexes under SPLUNK_HOME in the
# var/lib/splunk subdirectory.  This can be overridden
# here:
#
# SPLUNK_DB=/opt/splunk-home/var/lib/splunk
# Splunkd daemon name
SPLUNK_SERVER_NAME=Splunkd

# If SPLUNK_OS_USER is set, then Splunk service will only start
# if the 'splunk [re]start [splunkd]' command is invoked by a user who
# is, or can effectively become via setuid(2), $SPLUNK_OS_USER.
# (This setting can be specified as username or as UID.)
#
# SPLUNK_OS_USER
SPLUNK_OS_USER=splunk

in sudoers

splunk ALL=(ALL) NOPASSWD: /opt/splunk/bin/splunk restart
splunk ALL=(ALL) NOPASSWD: /opt/splunk/bin/splunk stop
splunk ALL=(ALL) NOPASSWD: /opt/splunk/bin/splunk start
splunk ALL=(ALL) NOPASSWD: /opt/splunk/bin/splunk status

Could it be an issue with SELinux?

Thanks in Advance

John

[isoutamo · Accepted solution]

Hi

As you expecting there could be something with SELinux. You could found reason from audit & selinux logs.

Why you are wanting to use init.d instead of systemd on recent linux? Currently systemd is the preferred way to start splunk on recent linux versions. My proposal is that switch to systemd and follow up these instructions Enable boot-start on machines that run systemd.

1. Chown all files to user splunk
2. Start first time as user splunk
3. Stop as user splunk
4. back to root/your id
5. enable bootstrap via sudo:  $SPLUNK_HOME/bin/splunk enable boot-start -user splunk -systemd-managed 1
6. Make test by booting server

r. Ismo

[jscraig2006]

Thanks for the quick reply.. I did attempt to use systemd to manage the boot-strap and couldn't get to work so I fell back to init.d. After following your steps which was simpler to follow than BOL. I was able to get it to start on boot with the splunk user using system managed so thank you very much! Now to find out how to get it to work with selinux in enforced mode. 

[isoutamo]

Good to hear that now it’s working.
If I recall right you must add some SELinux rights and also open firewall on host to get everything to work. But you could found those from google and local logs.