Unable to run "enable boot-start" as root user on RHEL 6.10

[batemanj]

Hi,

We installed/configured Splunk Universal forwarder 7.2.3 on hundreds of RHEL 6.10 in our environment and all of them working perfectly fine. There're few machines with selinux enabled with "permissive" mode. None of them are in "Enforcing" mode (thankfully).

Within these selinux (permissive mode) machines, only a few of them we were able to install/configure Splunk without any issues but on remaining machines we observed strange behavior.

After extracting the package, assigning ownership etc. we are able to set up the forward with sudo -H -u splunk /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --auto-ports --no-prompt and splunk starts successfully and communicates with Splunk Server but when I try to run /opt/splunkforwarder/bin/splunk enable boot-start -user splunk" as root user it simply stucks/hangs and the only way is to stop it by pressing ctrl-C or close the putty session. So basically root user is unable to run /opt/splunkforwarder/bin/splunk either for setup/start/stop, it just hangs. But both setup/start/stop work fine when we run as "splunk" user. With this behavior we are unable to run "enable boot-start" and weirdly this is happening only on few RHEL 6.10 servers with selinux permissive mode (not on all). Wondering if there's any additional security that causing this behavior of stopping root /opt/splunkforwarder/bin/splunk

I know we can do this boot-start part manually by creating a startup script in init.d which i would like to discuss later.

I appreciate any help, thanks!!

[codebuilder]

Ah, I see. I have a few thoughts/suggestions...

It's not necessary to modify /etc/init.d/splunk as you did in order for Splunk to run as the splunk user.
The init.d script invokes /opt/splunkforwarder/bin/splunk, which in turn reads values from /opt/splunkforwarder/etc/splunk-launch.conf. The value set for SPLUNK_OS_USER is (obviously) the user that Splunk will run as.

Also within splunk-launch.conf, ensure that the home path is set correctly: SPLUNK_HOME=/opt/splunkforwarder
And make sure that you don't have an OS environment variable overriding it, as that will take precedence:

13:28:51 # echo $SPLUNK_HOME
/opt/splunkforwarder
13:29:05 #:/

My suggestions for getting things back in order would be:
1. Stop (or hard kill) all running Splunk processes.
2. Revert your changes to /etc/init.d/splunk
3. Verify SPLUNK_HOME and SPLUNK_USER in splunk-launch.conf
4. Ensure user:group ownership of /opt/splunkforwarder: chown -RP splunk:splunk /opt/splunkforwarder
5. Revert any possible selinux changes (despite permissive mode): restorecon /opt/splunkforwarder/*
6. Verify the splunk user has home path set correctly:
13:37:18 # cat /etc/passwd |grep -i splunk
splunk:x:501:501:Splunk Server:/opt/splunkforwarder:/bin/bash
root@:/
8. As root, start Splunk via the init.d script: /etc/init.d/splunk start
9. Verify Splunk starts up under the splunk user: ps -ef |grep -i splunk

Try those steps and let me know your results.

[codebuilder]

Hrmm, that is odd behavior. What results do you get when you run the following?:

/etc/init.d/splunk status

service splunk status

Also, how did you verify that selinux is in permissive mode? What output do you get from the following?

getenforce

[codebuilder]

Also, do you have true root access, or are you sudo'ing to root from another user?

[batemanj]

Thanks agian codebuilder. I run as full root (not sudo).
As I can't run splunk as root, I added "sudo -H -u splunk" in /etc/init.d/splunk so that both commands you asked for are running fine. Prior to these changes, those commands did not work. Please see below:

[root@TESTSVR ~]# getenforce
Permissive [root@TESTSVR ~]#
[root@TESTSVR ~]# id uid=0(root)
gid=0(root)
groups=0(root),500(sapinst)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[root@TESTSVR ~]# [root@TESTSVR ~]#
time service splunk status Splunk
status: splunkd is running (PID:
5305). splunk helpers are running
(PIDs: 5311).

real 0m0.194s user 0m0.049s sys
0m0.121s [root@TESTSVR ~]#
[root@TESTSVR ~]# time
/etc/init.d/splunk status Splunk
status: splunkd is running (PID:
5305). splunk helpers are running
(PIDs: 5311).

real 0m0.147s user 0m0.048s sys
0m0.087s [root@TESTSVR ~]#
[root@TESTSVR ~]# time
/opt/splunkforwarder/bin/splunk status
^C

real 0m13.854s user 0m0.001s sys
0m0.002s [root@TESTSVR ~]#
[root@TESTSVR ~]# grep splunk
/etc/init.d/splunk |grep -i status
splunk_status() { /usr/bin/sudo -H
-u splunk bash -c '/opt/splunkforwarder/bin/splunk
status'
splunk_status [root@TESTSVR ~]#

[batemanj]

[root@TestBox etc]# grep SPLUNK_OS_USER /opt/splunkforwarder/etc/splunk-launch.conf |grep -vi "^#"
SPLUNK_OS_USER=splunk
[root@TestBox etc]# df -Ph /opt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/volgroup01-lv_root 13G 6.0G 6.1G 50% /
[root@TestBox etc]# ls -ltra /opt |grep -i splunk
drwxr-xr-x. 9 splunk splunk 4096 Oct 31 01:27 splunkforwarder
[root@TestBox etc]#

I ran "strace" while running "/opt/splunkforwarder/bin/splunk" as root and it reads sockets of winbind (weirdly) but when I run it as splunk user, it reads only "/opt/splunkforwarder/" files.

[codebuilder]

I've seen this behavior before running under init.d on new installs.

My first guess is that the SPLUNK_OS_USER is set to root, rather than splunk.
So when you try to start/stop using "sudo -Hu splunk", it's not being allowed as the process is owned by root.

Check the value of SPLUNK_OS_USER within /opt/splunkforwarder/etc/splunk-launch.conf, correct if necessary and cycle the daemon.

Note: You may have to also kill off any splunk processes running as root before it will be allowed.
Check that using:
ps -ef | grep -i splunk

[codebuilder]

Also, a chown -RP splunk:splunk /opt/splunkforwarder wouldn't hurt, and be certain that /opt has >5GB data free (the default).

[batemanj]

Thanks codebuilder for your reply.
I'm not sure if I conveyed my problem correctly here but we have problem running as root where everything works fine as "splunk user". Please please read my OP. however I checked all your pointers and all OK.

[root@TestBox etc]# grep
SPLUNK_OS_USER
/opt/splunkforwarder/etc/splunk-launch.conf
|grep -vi "^#" SPLUNK_OS_USER=splunk
[root@TestBox etc]# df -Ph /opt
Filesystem Size
Used Avail Use% Mounted on
/dev/mapper/volgroup01-lv_root 13G
6.0G 6.1G 50% / [root@TestBox etc]# ls -ltra /opt |grep -i splunk
drwxr-xr-x. 9 splunk splunk 4096
Oct 31 01:27 splunkforwarder
[root@TestBox etc]#

I ran "strace" while running "/opt/splunkforwarder/bin/splunk" as root and it reads sockets of winbind (weirdly) but when I run it as splunk user, it reads only "/opt/splunkforwarder/" files.

[woodcock]

Things changed with v7.? and started using systemd which doesn't work right in many cases. Try this:

splunk enable boot-start -systemd-managed 0

But be sure to go clean up the partial Systemd stuff that the previous command may have created!

[batemanj]

Hi woodcock,

Thanks for the reply but my RHEL version is still 6.10 (not RHEL 7) and there is no systemd in RHEL6.

Please notice that my problem is just that root is unable to run "/opt/splunkforwarder/bin/splunk" on few RHEL 6 machines.