Splunkforwarder rpm installation fails on default RHEL/CentOS 7.3 installation:
Can be reproduced with:
1) Default installation of CentOS 7.3 (e.g. with CentOS-7-x86_64-Minimal-1611.iso)
2) Install Splunk Universal Forwarder
rpm -i /tmp/splunkforwarder-6.6.0-1c4f3bbe1aea-linux-2.6-x86_64.rpm
useradd: cannot create directory /opt/splunkforwarder
complete3) Home directory /opt/splunkforwarder of "splunk" user is created but .bashrc for example is missing:
ls -ld /opt/splunkforwarder/.bashrc
ls: cannot access /opt/splunkforwarder/.bashrc: No such file or directory
5) SELinux denies (RHEL7 prevents home directories in /opt when installed with rpm scripts):
grep -ri denied /var/log/audit/audit.log
type=AVC msg=audit(1494600643.668:245): avc: denied { write } for pid=10460 comm="useradd" name="opt" dev="dm-0" ino=33646738 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:usr_t:s0 tclass=dir 6) Reason: Missing permission for useradd_t to write into usr_t (/opt Directory).
Is splunkforwarder supportet with a default installation of RHEL7? if yes, what is the official installation procedure for this rpm? Hope your solution is not "setenforce 0".
One should not need to to:
sudo setfacl -m u:splunk:rwx /opt
Not required at all. Please never do `chmod 0777 /dir` either.
This is not a DAC write perms issue in `/opt`. This is a SELinux issue. When the homedir creation "fails", files from `/etc/skel` are not copied into `/opt/splunk` so your `.bashrc` et al. are missing.
So there is a "Workaround" that can be used for this type of issue.
What is happening is with SELinux enabled there are context on specific Filesystem Directories which can prevent a Regular System user from making modifications. This is done "Out-of-the-Box" for any RHEL/CentOS Installation. You can take 2 Approaches for Systems Running SELinux.
1) The most common mitigation steps are to Disable SELinux. This is not always a "Good Idea" and may violate Company Security Policies. You can verify that SELinux Mode by running the following Command:
sestatus
[jthompson@jthompsonlin02 ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
So to Disable SELinux you would need to modify the following File: /etc/sysconfig/selinux. You would need to change the SELINUX=enforcing Line to match the setting that you are needing. In this case changing it from enforcing to disabled:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
If you complete the SELinux File changes then you will need to reboot the server to ensure that those changes take effect. Alternatively for a "temporary" workaround you could place SELinux into Permissive Mode which should allow the installation to complete as expected. The following provides a means of setting Permissive via the Command Line.
2) OR, You can disable SELinux from the command line using setenforce and the following is showing how to change from Enforcing to Permissive:
[jthompson@jthompsonlin02 ~]$ sudo setenforce 0
[sudo] password for jthompson:
[jthompson@jthompsonlin02 ~]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28
As you can see from the above printout that the Mode changed from Enforcing to Permissive when using: setenforce 0 -- to change this back you would use setenforce 1.
There is another Workaround that should resolve this issue as well if you are needing to have SELinux Enabled and do not have the ability to build out the required SELinux Context for the installation and running of Splunk. This Alternative method would be to use the setfacl (Set File ACL) Command and provide the Splunk User permissions on the /opt Directory. To make this type of change for the "Workaround" you would do the following:
[jthompson@jthompsonlin02 ~]$ sudo setfacl -m u:splunk:rwx /opt
[jthompson@jthompsonlin02 ~]$ getfacl /opt
getfacl: Removing leading '/' from absolute path names
# file: opt
# owner: root
# group: root
user::rwx
user:splunk:rwx
group::r-x
mask::rwx
other::r-x
Once you have completed the installation process you would want to remove the ACL from /opt using the following steps:
[jthompson@jthompsonlin02 ~]$ sudo setfacl -x u:splunk /opt
[jthompson@jthompsonlin02 ~]$ getfacl /opt
getfacl: Removing leading '/' from absolute path names
# file: opt
# owner: root
# group: root
user::rwx
group::r-x
mask::r-x
other::r-x
Any one of these "Alternative" Methods should allow the installation of Splunk to complete with SELinux enabled.
Thank for the info on this. Question: Once the Splunk Universal Forwarder is installed (using any of the above workarounds), can SELinux be set to back to enforce 1 (enabled) or will the Splunk Universal Forwarder have issues running in the default /opt/splunkforwarder directory? In other words, are the workarounds only needed for installation and not regular operation?
typically in *nix environments the product install dir and the home dir are not the same. i am using successfully using RHEL7+selinux with the universal forwarder and not having these issues. my splunk user home is /users/splunk and not /opt/splunkforwarder. my server admin team has given the splunk user permissions to write to /opt/splunkforwarder. further, using rpm is typically a root function, so this has to be resolved with your server admin team. whether they manage installing the package for you possibly via configuration management tool, or give the splunk user "sudo rpm" or similar.
I know there are several workarounds for this problem (different home directory, creating home dir manually before installation, disabling SELinux, modifying/extending SELinux policy, ...). Nevertheless, if Splunk supports RHEL7, I think it should be possible to install this RPM (with default install options) on a default RHEL7 system without errors which is not the case at the moment.
airlockOperationsApp
Splunk 7.x & 6.x are supported for use with RHEL 7. We have many customers that are using RHEL 7 with SELinux enabled and are not having any issues with the Installation or running of Splunk Universal Forwarder.
Please take a moment to work with your Server Admin Team to correct the Permissions issue that you have experienced. The following is a Clear Sign of Permissions Violations in the Linux FileSystem:
2) Install Splunk Universal Forwarder
useradd: cannot create directory /opt/splunkforwarder
complete
Please understand that depending on how you are attempting to install the Splunk Universal Forwarder will depend on the Permissions changes that might be needed. If you are attempting to install Splunk as a Non-Root User then you will need to provide that User with either Full Access to /opt to be able to Read/Write/Traverse through that directory. If the User Account that you are attempting to install Splunk as does not have permissions on the /opt Directory (which standard users should not be able to do as that is a "Root" FileSystem Directory and owned by Root).
The following is from: ls -al /
drwxrwxr-x+ 3 root root 19 Nov 5 2016 opt
As you can see the /opt Directory is owned by Root and as such NO Regular User will be able to modify that directory. You have a couple of options here to allow the needed Permissions to complete the Installation of Splunk Universal Forwarder as a "Regular" User:
1) Use the Set File ACL Command set to provide the required permissions to your User Account being used to install Splunk. This is something that your Server Administration Team would need to complete.
2) Change the permissions of /opt to be owned by the user that you are attempting to install Splunk Universal Forwarder as.
3) Change the Permissions of /opt from 755 to 777 to allow ANY User access to Read/Write/Traverse the /opt directory.
Please understand that the issue you are experiencing is Unique to your Environment as we have many Customers that are using RHEL 7 with SELinux "Out of the Box". I in fact have a Test System that is running CentOS 7 with SELinux Enabled and I am not having any issues with that Universal Forwarder. Also as advised by Koshyk if you are going to be using SELinux you or your Server Administration Team will need to setup the required SELinux Context for your Environment to allow your installation of the Splunk Universal Forwarder to function properly.
There is going to be work that you need to do to allow this installation to complete with your current environment. If you need SELinux enabled but need to be able to install or run Splunk you may be able to use one of the "Workarounds" that I provided previously.
To answer your question about re-enabling SELinux, Sure you can, but once again YOU or YOUR TEAM will be responsible for setting up the required SELinux Context to allow Splunk to function in your Environment. The GitHub Link provided by Koshyk will get you started down the right path for using SELinux and Splunk.
Welcome to the nightmare of SElinux.
1. Ensure your customer/manager knows SElinux out of box is not easy with Splunk. Ensure sufficient resourcing and maintenance overhead is allocated.
2. Please have a look into: https://github.com/doksu/selinux_policy_for_splunk and provide relevant permissions
Thanks. https://github.com/doksu/selinux_policy_for_splunk does not help for this problem. This is a policy file for the splunkforwarder process, rather than for the installation of the rpm package.
Is RHEL 7 with default targeted SELinux policy supported by splunkforwarder? If yes, i think somesone should fix the rpm.
Looks like permission issues...
I had the same problem.
The problem is a homedir in a non-standard location: https://www.unixmen.com/selinux-and-non-default-home-directory-locations/
$> ls -dlZ /home/robertg
drwx------. robertg robertg unconfined_u:object_r:user_home_dir_t:s0 /home/robertg
$> ls -dlZ /opt/splunk
drwxr-xr-x. splunk splunk unconfined_u:object_r:usr_t:s0 /opt/splunk
The above link explains in detail but the actual fix is:
#> semanage fcontext -a -e /home/robertg /opt/splunk
#This updates the labels of /opt/splunk, with the –R option doing it recursively for existing subdirectories.
#> restorecon –R /opt/splunk
#> ls -ldZ /opt/splunk
drwxr-xr-x. splunk splunk unconfined_u:object_r:user_home_dir_t:s0 /opt/splunk
If installing from RPM, see what creates the homedir:
$> rpm -q --scripts splunk
You can temporarily disable SELinux (not recommended):
#> setenforce 0
commands...
#> setenforce 1
Check any denied AVCs in `/var/log/audit/audit.log`.
Generally speaking: an app in its own dir like in /opt/appname is okay with SELinux as long as it writes there alone. Listening on ports below 1024 etc. requires additional SELinux work.
The current security in deploying to /opt/splunk using an interactive login account is wrong IMHO.
The account should be a system account (UID <1000) with no login like a classic UNIX daemon account. See `/etc/passwd` for examples.
I have tested the above and it works. Use Systemd to manage the service as `splunk:splunk` user, group respectively.