Splunk Add-on for Unix and Linux: rlog.sh not running, Amazon Linux $KERNEL not defined, audit.log has read permissions

[smitra_splunk]

Hi,

I'm faced with a weird issue where the /opt/splunk/etc/apps/Splunk_TA_nix/bin/rlog.sh does not do anything. I'm using Amazon Linux.

I ran an echo $KERNEL on the command prompt and there was nothing returned, hence the script code (below copied from the rlog.sh file) doesn't proceed further to where it will read the /var/log/audit/audit.log. I have confirmed that my rlog.sh has execute permissions by the splunk userid and the audit.log also has read ACL.
Shall I edit the script to just execute anyway by removing the check for the kernel type ? In that case, does "assertInvokerIsSuperuser" will also need to be removed.

SEEK_FILE=$SPLUNK_HOME/var/run/splunk/unix_audit_seekfile
AUDIT_FILE=/var/log/audit/audit.log

if [ "x$KERNEL" = "xLinux" ] ; then
    assertInvokerIsSuperuser
    assertHaveCommand service
    assertHaveCommandGivenPath /sbin/ausearch
    if [ -n "`service auditd status`" -a "$?" -eq 0 ] ; then
            if [ -e $SEEK_FILE ] ; then
                SEEK=`head -1 $SEEK_FILE`
            else
                SEEK=0
                echo "0" > $SEEK_FILE
            fi
            FILE_LINES=`wc -l $AUDIT_FILE  | cut -d " " -f 1`
            if [ $FILE_LINES -lt $SEEK ] ; then
                # audit file has wrapped
                SEEK=0
            fi
            awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
    fi
elif [ "x$KERNEL" = "xSunOS" ] ; then
    :
elif [ "x$KERNEL" = "xDarwin" ] ; then
    :
elif [ "x$KERNEL" = "xHP-UX" ] ; then
        :
elif [ "x$KERNEL" = "xFreeBSD" ] ; then
        :
fi

Any hint/direction is highly appreciated!

[Splunker]

And for assertHaveCommand() the following needs to be updated (line 47 of bin/common.sh from Splunk_TA_nix 6.0.2):

# # # /sbin/ is often absent in non-root users' PATH, and we want it for ifconfig(8)
PATH=$PATH:/sbin/

Ubuntu needs this, or it wont be able to find the "service" command when Splunk is running as non-root (splunk).

# # # Append path to help find commands when running as non-root, as the non-root paths are different
PATH=$PATH:/sbin/:/usr/sbin/

It's called from bin/rlog.sh as follows:

assertHaveCommand service

Cheers,

Chris.

[Splunker]

An update on this after a little digging..

In Splunk_TA_nix 6.0.2, It looks like the rlog.sh script is intended to run as root (the implication is Splunk runs as root), per the following check in common.sh:

assertInvokerIsSuperuser ()
{
    [ `id -u` -eq 0 ] && return
    echo "Must be superuser to run this script, quitting" > $TEE_DEST
    exit 1
}

If you enable debugging on rlog.sh (looks like it throws away this important output to /dev/null inside $TEE_DEST unless you have debug enabled):

sudo su - splunk
$SPLUNK_HOME/bin/splunk cmd $SPLUNK_HOME/etc/apps/Splunk_TA_nix/bin/rlog.sh --debug

You'll notice a debug file named debug--rlog.sh-- in the cwd which (unsurprisingly) says:

Must be superuser to run this script, quitting

Per the logic in assertInvokerIsSuperuser()

From an ES point-of-view, this is sub-optimal not only from a security standpoint (running Splunk as root), but the TA is of course designed to work from sourcetype=auditd keyed from the rlog.sh input, so it's not adding the value it could.

It's designed this way (i'd guess) because /var/log/audit/audit.log by way of ausearch is out-of-the-box only visible by root (without changes), but with proper unix/posix permissions setup, Splunk running as splunk, can ingest the file via ausearch, rlog.sh, etc.

It's debatable whether it's a security risk allowing a non-root user to read the audit.log file, but if you can't bring it up into Splunk to keep eyes on it, it's a relatively small risk to accept.

Anyway, just wanted to get to the bottom of why that was happening.. 🙂

PS: There's also a bug in assertHaveCommand() (at least on Ubuntu) i had to also work-around after assertInvokerIsSuperuser() to get it to work but i haven't yet found the root-cause for that, just a work-around, but looking into it..

Cheers,

Chris.

[cmakepeace_nfcu]

The KERNEL variable is not set within the OS but rather from another script within the Splunk_TA_nix. As the first line of the rlog.sh consists of:

. `dirname $0`/common.sh

Within this common.sh the KERNEL variable is set using:

# # # what OS is this?
KERNEL=`uname -s`

This is common across all Splunk_TA_nix scripts in how they run as many things are OS dependent. I would ensure that this common.sh is running properly as the need for commenting out the KERNEL if/elif segments shouldn't be needed.

[smitra_splunk]

I got it working by commenting out the following lines. Apparently Amazon Linux has all the dependencies satisfied as any other mainstream Linux, however the $KERNEL variable is not set (or may be need to be set when creating the EC2 instance, perhaps ?).

#if [ "x$KERNEL" = "xLinux" ] ; then
#     assertInvokerIsSuperuser
#     assertHaveCommand service
#     assertHaveCommandGivenPath /sbin/ausearch
     if [ -n "`service auditd status`" -a "$?" -eq 0 ] ; then
             if [ -e $SEEK_FILE ] ; then
                 SEEK=`head -1 $SEEK_FILE`
             else
                 SEEK=0
                 echo "0" > $SEEK_FILE
             fi
             FILE_LINES=`wc -l $AUDIT_FILE  | cut -d " " -f 1`
             if [ $FILE_LINES -lt $SEEK ] ; then
                 # audit file has wrapped
                 SEEK=0
             fi
             awk -v START=$SEEK -v OUTPUT=$SEEK_FILE 'NR>START { print } END { print NR > OUTPUT }' $AUDIT_FILE | tee $TEE_DEST | /sbin/ausearch -i 2>/dev/null | grep -v "^----"
     fi

# elif [ "x$KERNEL" = "xSunOS" ] ; then
#     :
# elif [ "x$KERNEL" = "xDarwin" ] ; then
#     :
# elif [ "x$KERNEL" = "xHP-UX" ] ; then
#         :
# elif [ "x$KERNEL" = "xFreeBSD" ] ; then
#         :
# fi