Automate installation and configuration of forwarder without a deployment server while using encrypted passwords and SSL

[christopher_stj]

I was looking for a way to fully automate the installation of Splunkforwarder without a deployment server and using encrypted passwords. Eventually I'm going to package this in an RPM.
After looking through several sources and submitting a ticket with Splunk I was able to locate a few files to do this.

This has been tested on RedHat Enterprise Linux 5.11/6.6/7.1 using splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm.
I use SSL and my own certificates for this setup.
I will call the Root CA "rootCA.cer", and the server certificate "splunkServerCert.pem". Both of these certificates should have been created when you installed the Splunk indexer. Splunk also supplies default ones.

You must first install the Splunk forwarder manually on one server to create the encrypted passwords and keys so you can copy them to other servers. See the Splunk documentation on how to do this.
Ensure to change the admin password from changeme to what you want. The encrypted admin password will be copied to the new servers.

$SPLUNK_HOME/bin/splunk edit user admin -password '<new password>'

When you run splunk commands the above password you changed will be the password you enter.

$SPLUNK_HOME/bin/splunk list forward-server

# Use the new password and not admin:changeme.

The only file I had to modify during the initial, manual install was /opt/splunkforwarder/etc/system/local/outputs.conf.
Here are the contents of /opt/splunkforwarder/etc/system/local/outputs.conf. The "server" and "sslPassword" have been <masked> for security.
The sslPassword is the password you entered for your splunkServerCert.pem.

/******************
[root@server15 ~]# cat /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = <F.Q.H.N>:9997
sslCertPath = $SPLUNK_HOME/etc/auth/splunkServerCert.pem
sslPassword = <Encrypted password>
sslRootCAPath = $SPLUNK_HOME/etc/auth/rootCA.cer
sslVerifyServerCert = true

/******************

After you install the Splunk forwarder copy $SPLUNK_HOME/etc/system/local/outputs.conf, $SPLUNK_HOME/etc/passwd, $SPLUNK_HOME/etc/auth/splunk.secret, $SPLUNK_HOME/etc/auth/rootCA.cer, $SPLUNK_HOME/etc/auth/splunkServerCert.pem, and splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm to a central location.

I uploaded the files to a web server.

Here is a quick script I put together to automate the install.

#!/bin/bash

wget --quiet -P /root http://<remote_server>/pub/outputs.conf http://<remote_server>/pub/passwd http://<remote_server>/pub/splunk.secret http://<remote_server>/pub/rootCA.cer http://<remote_server>/pub/splunkServerCert.pem http://<remote_server>/pub/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm

# Ensure the files exist or no point to continue. 
if [[ -f /root/outputs.conf ]] && [[ -f /root/passwd ]] && [[ -f /root/splunk.secret ]] && [[ -f /root/rootCA.cer ]] && [[ -f /root/splunkServerCert.pem ]] && [[ -f /root/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm ]] ; then 
  # To do: If selinux is turned on this will fail. Need to add selinux context settings here.
  yum -q -y localinstall /root/splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm

  # Just in case SELinux is troublesome. 
  if [[ -d /opt/splunkforwarder ]] ; then 
  cp /root/{rootCA.cer,splunkServerCert.pem,splunk.secret} /opt/splunkforwarder/etc/auth/
  cp /root/outputs.conf /opt/splunkforwarder/etc/system/local/
  cp /root/passwd /opt/splunkforwarder/etc/

  chown root:root /opt/splunkforwarder/etc/auth/{rootCA.cer,splunkServerCert.pem,splunk.secret} /opt/splunkforwarder/etc/system/local/outputs.conf /opt/splunkforwarder/etc/passwd
  chmod 400 /opt/splunkforwarder/etc/auth/{rootCA.cer,splunk.secret} 
  chmod 600 /opt/splunkforwarder/etc/system/local/outputs.conf /opt/splunkforwarder/etc/passwd /opt/splunkforwarder/etc/auth/splunkServerCert.pem

  # Customize Splunk. 
  /opt/splunkforwarder/bin/splunk start --accept-license &>/dev/null 
  /opt/splunkforwarder/bin/splunk enable boot-start &>/dev/null 
  /opt/splunkforwarder/bin/splunk add monitor /var/log/messages 
  /opt/splunkforwarder/bin/splunk add monitor /var/log/secure 
  /opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log 

  # Clean up. 
  rm -f /root/{outputs.conf,passwd,splunk.secret,rootCA.cer,splunkServerCert.pem,splunkforwarder-6.2.2-255606-linux-2.6-x86_64.rpm} 

  else 
    echo "ERROR: Missing /opt/splunkforwarder directory. It appears the RPM did not install correctly."
    yum -q -y erase splunkforwarder 
  fi 

else 
  echo "ERROR: Failed to install the splunk forwarder. Could not download the needed files from the web server."
fi

[RMcCurdyDOTcom]

cant post links so just search for freeload101 github for updated code

#!/bin/bash

########################## FUNC 
function UFYUM(){
cd /tmp
rpm -Uvh --nodeps `curl -s https://www.splunk.com/en_us/download/universal-forwarder.html\?locale\=en_us | grep -oP '"https:.*(?<=download).*x86_64.rpm"' |sed 's/\"//g' | head -n 1`
yum -y install splunkforwarder.x86_64
sleep 5

}

function UFDEB(){
cd /tmp
wget  `curl -s https://www.splunk.com/en_us/download/universal-forwarder.html\?locale\=en_us | grep -oP '"https:.*(?<=download).*amd64.deb"' |sed 's/\"//g' | head -n 1` -O amd64.deb
dpkg -i amd64.deb
sleep 5

}

function UFConf(){

mkdir -p /opt/splunkforwarder/etc/apps/nwl_all_deploymentclient/local/
cd /opt/splunkforwarder/etc/apps/nwl_all_deploymentclient/local/

cat <<EOF> /opt/splunkforwarder/etc/apps/nwl_all_deploymentclient/local/app.conf
[install]
state = enabled

[package]
check_for_updates = false

[ui]
is_visible = false
is_manageable = false
EOF

cat <<EOF> /opt/splunkforwarder/etc/apps/nwl_all_deploymentclient/local/deploymentclient.conf
[deployment-client]
phoneHomeIntervalInSecs = 60
[target-broker:deploymentServer]
targetUri = XXXXXXXXXXXXXXXXXXXXXXX:8089
EOF

cat <<EOF> /opt/splunkforwarder/etc/system/local/user-seed.conf
[user_info]
USERNAME = admin
PASSWORD = XXXXXXXXXXXXXXXXXXXXXXXX
EOF



/opt/splunkforwarder/bin/splunk cmd btool deploymentclient list --debug

/opt/splunkforwarder/bin/splunk start --accept-license
}

######################################################### MAIN 


# Check for RPM package managers
if command -v yum > /dev/null; then
	UFYUM
	UFConf
else
    echo "No YUM package manager found."
fi

# Check for DEB package managers
if command -v dpkg > /dev/null; then
	UFDEB
    UFConf
else
    echo "No DEB package manager found."
fi


 
 
 

[RMcCurdyDOTcom]

got nasty gram for posting links

search online for freeload101 github   in scripts Splunk_UniversalForwarder_Installer.bash

[christopher_stj · Accepted solution]

I have an update to my original post. I was able to package the Splunk installation as an RPM to automate the deployment to my clients through Red Hat Satellite.
I had to use the Splunk tar file because the RPM Splunk provides would throw checksum errors when installing on rhel7 through my Satellite server.
I could deploy to rhel6 using the Splunk RPM through Satellite though, but I chose not to, to be consistent. I tested this deployment on rhel6 and rhel7.
I masked some things for security , , . Before starting, you will need to do a complete manual install on one server to grab the files to be put into the RPM (SOURCES).
See the %install and %files sections for a complete list of files (outputs.conf, passwd, splunk.secret, inputs.conf, your certificates, splunkforwarder-*.gz. This setup uses the same certificate password and certificate on every server.
The RPM and directory aaastaging you see is a seperate RPM I created that sets up my envrionment variables for all of my custom RPMs.
The variable names are easy to determine what they are for so you can replace them as needed. The variable called CHROOT is to check if the RPM is being installed as part of the kickstart (CHROOT=yes, chroot environment) or not (CHROOT=no).
I am not the best RPM builder. If reinstalled, this RPM will fix what has changed. This is useful to baseline a server before deployments. Take from this what you can.

#-------------------------------------------------------------------------#
# Define variables for later use
%define name            aaasplunk
%define version         11.0
%define release         10%{?dist}
%define stagedir        /etc/aaastaging/aaasplunk

#-------------------------------------------------------------------------#
# Package information
Packager:               <Your name and email goes here>
Summary:                Splunk setup and configuration for RHEL %{rhel}.
Name:                   %{name}
Version:                %{version}
Release:                %{release}
BuildArch:              x86_64
Group:                  aaa
License:                zzz
Url:                    http://www.zzz.com
Vendor:                 <Your vendor here>
Source0:                %{name}-%{version}.tgz
Buildroot:              %{_tmppath}/%{name}-%{version}-buildroot
BuildRequires:          coreutils
# rhel6 specific.
%{?el6:Requires:        coreutils, findutils, e2fsprogs, grep, gawk, glibc-common, shadow-utils, tar, sed, initscripts, policycoreutils, libselinux-utils, rpm, procps, aaastaging, util-linux-ng}
# rhel7 specific.
%{?el7:Requires:        coreutils, findutils, e2fsprogs, grep, gawk, glibc-common, shadow-utils, tar, sed, initscripts, policycoreutils, libselinux-utils, rpm, procps-ng, aaastaging, util-linux, systemd}

#-------------------------------------------------------------------------#
# Package Description
%description
This rpm will setup and configure Splunk using the Splunk zip file splunkforwarder-x.x.x-xxxxxx-Linux-x86_64.gz.

#-------------------------------------------------------------------------#
# What to do in prep for building
%prep
%setup -q

#-------------------------------------------------------------------------#
# Build process (not really needed most times)
%build

#-------------------------------------------------------------------------#
## Package install process
%install
/bin/rm -rf %{buildroot}
/bin/mkdir -p -m 0700 %{buildroot}%{stagedir}/certs/
# Use a temporary location to put the files until the splunkforwarder tar file is exploded.
/usr/bin/install -m 0600 etc/aaastaging/aaasplunk/outputs.conf %{buildroot}%{stagedir}/
/usr/bin/install -m 0600 etc/aaastaging/aaasplunk/passwd %{buildroot}%{stagedir}/
/usr/bin/install -m 0400 etc/aaastaging/aaasplunk/splunk.secret %{buildroot}%{stagedir}/
/usr/bin/install -m 0700 etc/aaastaging/aaasplunk/splunkforwarder-*.gz %{buildroot}%{stagedir}/
# Additional monitored files and directories should be put in inputs.conf.
/usr/bin/install -m 0600 etc/aaastaging/aaasplunk/inputs.conf %{buildroot}%{stagedir}/
for cert in etc/aaastaging/aaasplunk/certs/*.cer ; do /usr/bin/install -m 0400 $cert %{buildroot}%{stagedir}/certs/ ; done
for cert in etc/aaastaging/aaasplunk/certs/*.pem ; do /usr/bin/install -m 0600 $cert %{buildroot}%{stagedir}/certs/ ; done

#-------------------------------------------------------------------------#
## Pre install scripts
%pre
  # Setup variables.
  . /etc/aaastaging/var.conf
  error="no"

  # These variables are required in this script. Do not continue without them.
  [[ -z $IP ]] && { /bin/echo 'FATAL: There was a problem sourcing the IP from /etc/aaastaging/var.conf. Exiting...' >&2 ; error="yes" ; }
  [[ -z $SITE ]] && { /bin/echo 'FATAL: There was a problem sourcing the SITE from /etc/aaastaging/var.conf. Exiting...' >&2 ; error="yes" ; }
  [[ -z $SPLUNK_HN ]] && { /bin/echo 'FATAL: There was a problem sourcing the SPLUNK_HN from /etc/aaastaging/var.conf. Exiting...' >&2 ; error="yes" ; }
  [[ -z $OS_VERSION ]] && { /bin/echo 'FATAL: There was a problem sourcing the OS_VERSION from /etc/aaastaging/var.conf. Exiting...' >&2 ; error="yes" ; }
  [[ -z $CHROOT ]] && { /bin/echo 'FATAL: There was a problem sourcing the CHROOT from /etc/aaastaging/var.conf. Exiting...' >&2 ; error="yes" ; }

  # Check for existing splunkforwarder RPM install.
  /bin/rpm -q splunkforwarder &>/dev/null && { /bin/echo 'FATAL: Found an existing splunkforwarder installation. You must remove it first. Exiting...' >&2 ; error="yes" ; }

  # Do not install the forwarder on the Splunk server.
  /bin/rpm -q splunk &>/dev/null && { /bin/echo 'FATAL: You cannot install the splunkforwarder on the Splunk server. Exiting...' >&2 ; error="yes" ; }

  # This is my site specific UID/GID. Use the number you want.
  # If the user accounts or groups already exist ensure they have the correct UID/GID.
  if /bin/grep '^splunk:' /etc/passwd &>/dev/null ; then
    [[ $(/usr/bin/id -u splunk) -ne 1111 ]] && { /bin/echo 'FATAL: The UID for splunk must be 1111. Exiting...' >&2 ; error="yes" ; }
    [[ $(/usr/bin/id -g splunk) -ne 1111 ]] && { /bin/echo 'FATAL: The GID for splunk must be 1111. Exiting...' >&2 ; error="yes" ; }
  fi

  if /bin/grep ':1111:' /etc/passwd &>/dev/null ; then
    [[ $(/bin/awk -F: '$3 ~ /^1111$/ {print $1}' /etc/passwd) != "splunk" ]] && { /bin/echo 'FATAL: UID 1111 is in use by another account. Exiting...' >&2 ; error="yes" ; }
  fi

  if /bin/grep '^splunk:' /etc/group &>/dev/null ; then
   [[ $(/bin/awk -F: '$1 ~ /^splunk$/ {print $3}' /etc/group) -ne 1111 ]] && { /bin/echo 'FATAL: The GID for splunk must be 1111. Exiting...' >&2 ; error="yes" ; }
  fi

  if /bin/grep ':1111:' /etc/group &>/dev/null ; then
   [[ $(/bin/awk -F: '$3 ~ /^1111$/ {print $1}' /etc/group) != "splunk" ]] && { /bin/echo 'FATAL: GID 1111 is in use by another group. Exiting...' >&2 ; error="yes" ; }
  fi

  [[ $error = "yes" ]] && exit 1

  # Remove immutable bit for files.
  [[ -d %{stagedir} ]] && /bin/find %{stagedir}/ -type f -exec /usr/bin/chattr -i {} \;

  exit 0

#-------------------------------------------------------------------------#
## Post install scripts.
%post
  # Run these steps for new install and upgrade.

  # RHEL-06-000509 LOW. The system must forward audit records to the syslog service.
  # RHEL-06-000136 CAT II. The operating system must back up audit records on an organization defined frequency onto a different system or media than the system being audited.
  # RHEL-06-000137 CAT II. The operating system must support the requirement to centrally manage the content of audit records generated by organization defined information system components.
  # We will use splunk instead of syslog.

  # Setup variables.
  . /etc/aaastaging/var.conf
  # You must set SElinux to permissive or the install will fail.
  se_status="$(/usr/sbin/getenforce)"
  restart_splunk="no"
  [[ "$se_status" = "Enforcing" ]] && /usr/sbin/setenforce permissive &>/dev/null

if [[ $1 -eq 1 ]] ; then
  # New install.

  # Check for a manual install and remove it.
  kill_list="$(/bin/ps -ef | /bin/grep splunk | /bin/grep -v aaasplunk | /bin/grep -v grep | /bin/awk '{print $2}')"
  if [[ -d /opt/splunkforwarder ]] || [[ -f /var/spool/mail/splunk ]] || [[ -n "$kill_list" ]] ; then
    /bin/echo 'INFO: A manual Splunk installation or unclean uninstall was detected. Running cleanup procedures...' >&2
    [[ -f /opt/splunkforwarder/bin/splunk && $CHROOT = "no" ]] && /opt/splunkforwarder/bin/splunk stop &>/dev/null
    [[ -n "$kill_list" ]] && { /usr/bin/pkill -u splunk &>/dev/null ; /usr/bin/pkill -u splunk &>/dev/null ; /bin/kill -9 $kill_list &>/dev/null ; }
    [[ -d /opt/splunkforwarder ]] && /bin/rm -rf /opt/splunkforwarder 2>/dev/null
    [[ -f /var/spool/mail/splunk ]] && /bin/rm -rf /var/spool/mail/splunk 2>/dev/null
    restart_splunk="yes"
  fi
fi


  # Run these steps for new install and upgrade.

  # Create the splunk user and group otherwise the installer will use the next highest uid/gid which we don't want.
  /usr/bin/getent group splunk &>/dev/null || { /bin/echo 'INFO: Creating the splunk group...' >&2 ; /usr/sbin/groupadd -g 1111 splunk &>/dev/null ; restart_splunk="yes" ; }
  /usr/bin/getent passwd splunk &>/dev/null || { /bin/echo 'INFO: Creating the splunk user...' >&2 ; /usr/sbin/useradd -d /opt/splunkforwarder -c "Splunk Server" -g splunk -M -s /bin/bash -u 1111 splunk &>/dev/null ; restart_splunk="yes" ; }

  # Install the newest (by timestamp) Splunkforwarder we have in %{stagedir}.
  # you can package multiple versions of the Splunk forwarder in the RPM and it will use the latest.
  path_new_sf="$(/bin/ls -tr %{stagedir}/splunkforwarder-*.gz | /usr/bin/tail -1)"
  version_new_sf="$(/bin/basename $path_new_sf | /bin/awk -F"-" '{print $2"-"$3}')"
  version_current_sf="$(/opt/splunkforwarder/bin/splunk -version 2>/dev/null | /bin/awk '{print $4"-"$6}' | /bin/awk -F")" '{print $1}')"
  if [[ "$version_new_sf" != "$version_current_sf" ]] ; then
    /bin/echo 'INFO: A newer (by timestamp) splunkforwarder was found. Installing it...' >&2
    [[ -f /opt/splunkforwarder/bin/splunk && $CHROOT = "no" ]] && /opt/splunkforwarder/bin/splunk stop &>/dev/null
    [[ -d /opt/splunkforwarder ]] && /bin/rm -rf /opt/splunkforwarder 2>/dev/null
    /bin/tar -C /opt/ -xf $path_new_sf
    [[ "$se_status" != "Disabled" ]] && /sbin/restorecon -R /opt/splunkforwarder
    restart_splunk="yes"
  fi

  # /opt/splunkforwarder/etc/system/local/inputs.conf
  # In inputs.conf, additional monitoring is built dynamically in this rpm,
  # therefore %{stagedir}/inputs.conf will never be the same as /opt/splunkforwarder/etc/system/local/inputs.conf and will always trigger a rebuild of /opt/splunkforwarder/etc/system/local/inputs.conf.
  # To avoid this behavior, build a temporary file as if it was the real file and compare it to the real file. If they differ copy the temporary file in place of the real file.
  /bin/cp -af %{stagedir}/inputs.conf /opt/splunkforwarder/etc/system/local/inputs.conf.temp
  [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/system/local/inputs.conf.temp
  /bin/sed -i "s/^host =.*/host = ${HOSTNAME}/" /opt/splunkforwarder/etc/system/local/inputs.conf.temp

  # ** All additional monitoring must be appended to inputs.conf before starting the Splunk service. **
  # Server-specific customization for inputs.conf.
  # The customization is triggered from the hostname. Add your specific hosts here.

  # Specific to JBoss EAP 6.4+.
  if [[ ${HOSTNAME/jboss} != ${HOSTNAME} ]] ; then
    /bin/grep 'monitor:///usr/share/jbossas/domain/log/servers/Service/server.log' /opt/splunkforwarder/etc/system/local/inputs.conf.temp &>/dev/null || /bin/echo -e "\n[monitor:///usr/share/jbossas/domain/log/servers/Service/server.log]\ndisabled = false" >> /opt/splunkforwarder/etc/system/local/inputs.conf.temp
    /bin/grep 'monitor:///usr/share/jbossas/domain/log/servers/Web/server.log' /opt/splunkforwarder/etc/system/local/inputs.conf.temp &>/dev/null || /bin/echo -e "\n[monitor:///usr/share/jbossas/domain/log/servers/Web/server.log]\ndisabled = false" >> /opt/splunkforwarder/etc/system/local/inputs.conf.temp
  fi

  # Use the cksum of the temp file compared against the live file.
  if [[ "$(/usr/bin/cksum /opt/splunkforwarder/etc/system/local/inputs.conf.temp 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/system/local/inputs.conf 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
    if [[ -f %{stagedir}/inputs.conf.orig ]] ; then /bin/cp -af /opt/splunkforwarder/etc/system/local/inputs.conf %{stagedir}/inputs.conf.backup 2>/dev/null ; else /bin/cp -af /opt/splunkforwarder/etc/system/local/inputs.conf %{stagedir}/inputs.conf.orig 2>/dev/null ; fi
    /bin/echo 'INFO: Installing /opt/splunkforwarder/etc/system/local/inputs.conf...' >&2
    /bin/mv -f /opt/splunkforwarder/etc/system/local/inputs.conf.temp /opt/splunkforwarder/etc/system/local/inputs.conf
    restart_splunk="yes"
  else
    /bin/rm -f /opt/splunkforwarder/etc/system/local/inputs.conf.temp
  fi


  # /opt/splunkforwarder/etc/system/local/outputs.conf
  # In outputs.conf, the parameter server is built dynamically in this rpm,
  # therefore %{stagedir}/outputs.conf will never be the same as /opt/splunkforwarder/etc/system/local/outputs.conf and will always trigger a rebuild of /opt/splunkforwarder/etc/system/local/outputs.conf.
  # To avoid this behavior, build a temporary file as if it was the real file and compare it to the real file. If they differ copy the temporary file in place of the real file.
  /bin/cp -af %{stagedir}/outputs.conf /opt/splunkforwarder/etc/system/local/outputs.conf.temp
  [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/system/local/outputs.conf.temp
  /bin/sed -i "s/^server =.*/server = ${SPLUNK_HN}:9997/" /opt/splunkforwarder/etc/system/local/outputs.conf.temp

  # Use the cksum of the temp file compared against the live file.
  if [[ "$(/usr/bin/cksum /opt/splunkforwarder/etc/system/local/outputs.conf.temp 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/system/local/outputs.conf 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
    if [[ -f %{stagedir}/outputs.conf.orig ]] ; then /bin/cp -af /opt/splunkforwarder/etc/system/local/outputs.conf %{stagedir}/outputs.conf.backup 2>/dev/null ; else /bin/cp -af /opt/splunkforwarder/etc/system/local/outputs.conf %{stagedir}/outputs.conf.orig 2>/dev/null ; fi
    /bin/echo 'INFO: Installing /opt/splunkforwarder/etc/system/local/outputs.conf...' >&2
    /bin/mv -f /opt/splunkforwarder/etc/system/local/outputs.conf.temp /opt/splunkforwarder/etc/system/local/outputs.conf
    restart_splunk="yes"
  else
    /bin/rm -f /opt/splunkforwarder/etc/system/local/outputs.conf.temp
  fi

  # /opt/splunkforwarder/etc/passwd
  if [[ "$(/usr/bin/cksum %{stagedir}/passwd 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/passwd 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
    if [[ -f %{stagedir}/passwd.orig ]] ; then /bin/cp -af /opt/splunkforwarder/etc/passwd %{stagedir}/passwd.backup 2>/dev/null ; else /bin/cp -af /opt/splunkforwarder/etc/passwd %{stagedir}/passwd.orig 2>/dev/null ; fi
    /bin/echo 'INFO: Installing /opt/splunkforwarder/etc/passwd...' >&2
    /bin/cp -af %{stagedir}/passwd /opt/splunkforwarder/etc/
    [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/passwd
    restart_splunk="yes"
  fi

  # /opt/splunkforwarder/etc/auth/splunk.secret
  if [[ "$(/usr/bin/cksum %{stagedir}/splunk.secret 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/auth/splunk.secret 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
    if [[ -f %{stagedir}/splunk.secret.orig ]] ; then /bin/cp -af /opt/splunkforwarder/etc/auth/splunk.secret %{stagedir}/splunk.secret.backup 2>/dev/null ; else /bin/cp -af /opt/splunkforwarder/etc/auth/splunk.secret %{stagedir}/splunk.secret.orig 2>/dev/null ; fi
    /bin/echo 'INFO: Installing /opt/splunkforwarder/etc/auth/splunk.secret...' >&2
    /bin/cp -af %{stagedir}/splunk.secret /opt/splunkforwarder/etc/auth/
    [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/auth/splunk.secret
    restart_splunk="yes"
  fi

  # Copy the environment-specific root CA and server certs.
  if [[ -f %{stagedir}/certs/${SITE}-rootCA.cer ]] ; then
    if [[ "$(/usr/bin/cksum %{stagedir}/certs/${SITE}-rootCA.cer 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/auth/${SITE}-rootCA.cer 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
      /bin/echo "INFO: Installing /opt/splunkforwarder/etc/auth/${SITE}-rootCA.cer..." >&2
      /bin/cp -af %{stagedir}/certs/${SITE}-rootCA.cer /opt/splunkforwarder/etc/auth/
      [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/auth/${SITE}-rootCA.cer
      restart_splunk="yes"
    fi
  else
    /bin/echo 'ERROR: Could not find the rootCA.cer for this environment in %{stagedir}/certs/.' >&2
  fi

  if [[ -f %{stagedir}/certs/${SITE}-serverCert.pem ]] ; then
    if [[ "$(/usr/bin/cksum %{stagedir}/certs/${SITE}-serverCert.pem 2>/dev/null | /bin/awk '{print $1}')" -ne "$(/usr/bin/cksum /opt/splunkforwarder/etc/auth/${SITE}-serverCert.pem 2>/dev/null | /bin/awk '{print $1}')" ]] ; then
      /bin/echo "INFO: Installing /opt/splunkforwarder/etc/auth/${SITE}-serverCert.pem..." >&2
      /bin/cp -af %{stagedir}/certs/${SITE}-serverCert.pem /opt/splunkforwarder/etc/auth/
      [[ $se_status != "Disabled" ]] && /sbin/restorecon /opt/splunkforwarder/etc/auth/${SITE}-serverCert.pem
      restart_splunk="yes"
    fi
  else
    /bin/echo 'ERROR: Could not find the serverCert.pem for this environment in %{stagedir}/certs/.' >&2
  fi

  # Remove extraneous certs.
  /bin/find %{stagedir}/certs/ -type f -not -name "${SITE}*" -delete

  # You will need to tweek this for your environment.
  # Update /etc/audit/audit.rules because Splunk will update audit.log as it reads it creating a compound affect.
  if [[ $OS_VERSION -eq 6 ]] ; then
    if ! /bin/grep '^-w /var/log/audit -p wax -k aaa-audit$' /etc/audit/audit.rules &>/dev/null ; then
      /bin/echo 'INFO: Updating /etc/audit/audit.rules for Splunk to function properly...' >&2
      [[ $CHROOT = "no" ]] && /sbin/service auditd stop 1>/dev/null
      if /bin/grep '^-w /var/log/audit -k aaa-audit$' /etc/audit/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit -k aaa-audit$/-w \/var\/log\/audit -p wax -k aaa-audit/' /etc/audit/audit.rules
      elif /bin/grep '^-w /var/log/audit' /etc/audit/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit/-w \/var\/log\/audit -p wax -k aaa-audit/' /etc/audit/audit.rules
      else
        /bin/echo '-w /var/log/audit -p wax -k aaa-audit' >> /etc/audit/audit.rules
      fi
      [[ $CHROOT = "no" ]] && /sbin/service auditd start 1>/dev/null
    fi
  fi

  # You will need to tweek this for your environment.
  # Update /etc/audit/rules.d/audit.rules because Splunk will update audit.log as it reads it creating a compound affect.
  if [[ $OS_VERSION -eq 7 ]] ; then
    if ! /bin/grep '^-w /var/log/audit -p wax -k var_log_audit$' /etc/audit/rules.d/audit.rules &>/dev/null ; then
      /bin/echo 'INFO: Updating /etc/audit/rules.d/audit.rules for Splunk to function properly...' >&2
      [[ $CHROOT = "no" ]] && { /usr/bin/systemctl --quiet is-active auditd.service && /sbin/service auditd stop 1>/dev/null ; }
      if /bin/grep '^-w /var/log/audit -k var_log_audit$' /etc/audit/rules.d/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit -k var_log_audit$/-w \/var\/log\/audit -p wax -k var_log_audit/' /etc/audit/rules.d/audit.rules
      elif /bin/grep '^-w /var/log/audit' /etc/audit/rules.d/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit/-w \/var\/log\/audit -p wax -k var_log_audit/' /etc/audit/rules.d/audit.rules
      else
        /bin/echo '-w /var/log/audit -p wax -k var_log_audit' >> /etc/audit/rules.d/audit.rules
      fi
      [[ $CHROOT = "no" ]] && /sbin/service auditd start 2>/dev/null
    fi
  fi

  # Start and enable Splunk.
  if [[ $restart_splunk = "yes" ]] ; then
    [[ $CHROOT = "no" ]] && { /opt/splunkforwarder/bin/splunk stop &>/dev/null ; /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes 1>/dev/null ; }
    /opt/splunkforwarder/bin/splunk enable boot-start --accept-license &>/dev/null
  fi

  # Re-enable SElinux.
  [[ "$se_status" = "Enforcing" ]] && /usr/sbin/setenforce enforcing

  # Make all files immutable.
  [[ -d %{stagedir} ]] && /bin/find %{stagedir}/ -type f -exec /usr/bin/chattr +i {} \;

  exit 0

#-------------------------------------------------------------------------#
## Pre uninstall scripts.
%preun
if [[ $1 -eq 0 ]] ; then
  # Complete uninstall.

  # Setup variables.
  . /etc/aaastaging/var.conf

  # These variables are required in this script.
  [[ -z $IP ]] && /bin/echo 'FATAL: There was a problem sourcing the IP from /etc/aaastaging/var.conf. Errors will result. Continuing...' >&2
  [[ -z $OS_VERSION ]] && /bin/echo 'FATAL: There was a problem sourcing the OS_VERSION from /etc/aaastaging/var.conf. Errors will result. Continuing...' >&2
  [[ -z $CHROOT ]] && /bin/echo 'FATAL: There was a problem sourcing the CHROOT from /etc/aaastaging/var.conf. Errors will result. Continuing...' >&2

  # Remove immutable bit for files.
  [[ -d %{stagedir} ]] && /bin/find %{stagedir}/ -type f -exec /usr/bin/chattr -i {} \;
fi

exit 0

#-------------------------------------------------------------------------#
## Post uninstall scripts.
%postun
if [[ $1 -eq 0 ]] ; then
  # Complete uninstall.

  # Setup variables.
  . /etc/aaastaging/var.conf

  # Disable boot start.
  [[ -f /opt/splunkforwarder/bin/splunk ]] && /opt/splunkforwarder/bin/splunk disable boot-start &>/dev/null

  # Uninstall Splunk.
  [[ -f /opt/splunkforwarder/bin/splunk && $CHROOT = "no" ]] && /opt/splunkforwarder/bin/splunk stop &>/dev/null
  kill_list="$(/bin/ps -ef | /bin/grep splunk | /bin/grep -v aaasplunk | /bin/grep -v grep | /bin/awk '{print $2}')"
  [[ -n "$kill_list" ]] && { /usr/bin/pkill -u splunk &>/dev/null ; /usr/bin/pkill -u splunk &>/dev/null ; /bin/kill -9 $kill_list &>/dev/null ; }
  /usr/bin/getent passwd splunk &>/dev/null && /usr/sbin/userdel -rf splunk &>/dev/null
  /usr/bin/getent group splunk &>/dev/null && /usr/sbin/groupdel splunk &>/dev/null
  [[ -d /opt/splunkforwarder ]] && /bin/rm -rf /opt/splunkforwarder 2>/dev/null
  [[ -f /var/spool/mail/splunk ]] && /bin/rm -rf /var/spool/mail/splunk

  # Restore /etc/audit/audit.rules.
  if [[ $OS_VERSION -eq 6 ]] ; then
    if ! /bin/grep '^-w /var/log/audit -k aaa-audit$' /etc/audit/audit.rules &>/dev/null ; then
      [[ $CHROOT = "no" ]] && /sbin/service auditd stop 1>/dev/null
      if /bin/grep '^-w /var/log/audit -p wax -k aaa-audit$' /etc/audit/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit -p wax -k aaa-audit$/-w \/var\/log\/audit -k aaa-audit/' /etc/audit/audit.rules
      elif /bin/grep '^-w /var/log/audit' /etc/audit/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit/-w \/var\/log\/audit -k aaa-audit/' /etc/audit/audit.rules
      else
        /bin/echo 'WARNING: Could not restore '-w \/var\/log\/audit -k aaa-audit' in /etc/audit/audit.rules.' >&2
      fi
      [[ $CHROOT = "no" ]] && /sbin/service auditd start 1>/dev/null
    fi
  fi

  if [[ $OS_VERSION -eq 7 ]] ; then
    if ! /bin/grep '^-w /var/log/audit -p wax -k var_log_audit$' /etc/audit/rules.d/audit.rules &>/dev/null ; then
      [[ $CHROOT = "no" ]] && { /usr/bin/systemctl --quiet is-active auditd.service && /sbin/service auditd stop 1>/dev/null ; }
      if /bin/grep '^-w /var/log/audit -k var_log_audit$' /etc/audit/rules.d/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit -k var_log_audit$/-w \/var\/log\/audit -p wax -k var_log_audit/' /etc/audit/rules.d/audit.rules
      elif /bin/grep '^-w /var/log/audit' /etc/audit/rules.d/audit.rules &>/dev/null ; then
        /bin/sed -i 's/^-w \/var\/log\/audit/-w \/var\/log\/audit -p wax -k var_log_audit/' /etc/audit/rules.d/audit.rules
      else
        /bin/echo '-w /var/log/audit -p wax -k var_log_audit' >> /etc/audit/rules.d/audit.rules
      fi
      [[ $CHROOT = "no" ]] && /sbin/service auditd start 2>/dev/null
    fi
  fi

  # Clean up.
  [[ -d %{stagedir} ]] && /bin/rm -rf %{stagedir}
fi

exit 0

#-------------------------------------------------------------------------#
# Files section, what do we install and how we keep track of it
%files
%defattr(0400,root,root)
%config %attr(0600,root,root) %{stagedir}/outputs.conf
%config %attr(0600,root,root) %{stagedir}/passwd
%config %attr(0400,root,root) %{stagedir}/splunk.secret
%config %attr(0600,root,root) %{stagedir}/inputs.conf
%{stagedir}/certs/
%{stagedir}/splunkforwarder-*.gz
%dir %{stagedir}/

#-------------------------------------------------------------------------#
# Cleanup after build
%clean
/bin/rm -rf %{buildroot}

#-------------------------------------------------------------------------#
# Changelog
%changelog
* Mon Dec 21 2015 <Masked for security> 11.0-10
- Add your notes here.