All Apps and Add-ons

Why is the S.o.S - Splunk on Splunk auditd.service giving errors when running rlog.sh in the Splunk Add-on for Unix and Linux?

Communicator

I am getting errors in SOS as below. I think that the Splunk_TA_nix app is causing them when running rlog.sh:

Redirecting to /bin/systemctl status  auditd.service
type=USER_ACCT msg=audit(10/19/2015 15:50:01.877:15323) : pid=11620 uid=root auid=unset ses=unset subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(10/19/2015 15:50:01.877:15324) : pid=11620 uid=root auid=unset ses=unset subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(10/19/2015 15:50:01.878:15325) : pid=11620 uid=root subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=root old-ses=4294967295 ses=2076 res=yes
type=USER_START msg=audit(10/19/2015 15:50:01.889:15326) : pid=11620 uid=root auid=root ses=2076 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(10/19/2015 15:50:01.889:15327) : pid=11620 uid=root auid=root ses=2076 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(10/19/2015 15:50:01.901:15328) : pid=11620 uid=root auid=root ses=2076 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(10/19/2015 15:50:01.903:15329) : pid=11620 uid=root auid=root ses=2076 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct=root exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success'

Sometimes, just the first line is shown and sometimes, the whole of the log is shown. When I log in and run the script as root, I get the issue, but when I login as splunk and run the script, I get nothing.

I checked the scripts and made sure that the WHOLE of the /opt/splunk directory is owned by splunk:splunk with 755 (as I thought that it was a permissions issue).

What am I doing wrong and is there something else that I need to install to get it working? There seem to be other, similar errors around.

I am running CentOS 7.0 with Splunk 6.2.3.

Kindest regards,

BlueSocket

Explorer

This is how (I think) I fixed this error:

Change this line in rlog.sh from:
if [ -n "service auditd status" -a "$?" -eq 0 ] ; then
To:
if [ -n "service auditd status 2> /dev/null" -a "$?" -eq 0 ] ; then

Why this works?

Because on rhel7 the redirecting to systemctl comment is sent to stderr which splunk interprets as an error it should log under the ExecProcessor in splunkd.log

Engager

Thanks for posting this - this worked perfectly for me.

0 Karma

Splunk Employee
Splunk Employee

when in doubt update... there have been many fixes, including some to rlog.sh. http://docs.splunk.com/Documentation/UnixAddOn/5.2.0/User/Releasenotes

0 Karma

Communicator

I think that I have worked out what is causing the error - the script is executing and the "service auditd status" line is causing the following, because bash on CentoOS 7 reinterpreting the command and redirecting it to the following and the command returns an error:

Redirecting to /bin/systemctl status  auditd.service

I think that the error value that is being returned is not REALLY an error, but a Warning, instead.

I see that the version of Splunk_TA_nix is 5.1.2 and that there is a revision (5.2.0) out. Will that fix this issue?

0 Karma

Splunk Employee
Splunk Employee

we are seeing the same issue on redhat ver 3.10. it looks like the rlog.sh script is not expecting the os to reroute the service call to a 'different' service, so it writes an error. i have contacted the team responsible for development on the unix TA and opened a bug with them.

0 Karma

Communicator

OK, wow! It looks like I have a genuine bug and it is not my mistake!

It might be an idea if we talk direct, as this is just one of many, similar, errors from the Splunk_TA_nix App, I think.

0 Karma

Splunk Employee
Splunk Employee

any progress on this? I'm experiencing the same thing here.

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!