All Apps and Add-ons

Why does Splunk_TA_nix rlog.sh cause huge amount of duplicate audit events on Ubuntu?

dshakespeare_sp
Splunk Employee
Splunk Employee

Customer reported that after enabling rlog.sh on Ubuntu Universal Forwarder they were getting 4 x the number of events they were expecting. From a 500,000 line audit.log, they received over 2 million events in about 10 minutes. Further investigation showed that events were being duplicated by a factor of 4.

dshakespeare_sp
Splunk Employee
Splunk Employee

Audit.log is read by a scripted input ( rlog.sh ) which runs every 60 secs
The script keeps is own seek_pointer ( unix_audit_seekfile ) in $SPLUNK_HOME/var/run/splunk
The problem is caused by the fact that the script runs under #!/bin/sh
Part of the code tests for the existence of unix_audit_seekfile with [ -a $SEEK_FILE ]
If the file does not exist, we create a new pointer " 0 "

On most systems /bin/sh is a link to /bin/bash BUT on Ubuntu it is a link to /bin/dash
[ -a $SEEK_FILE ] will fail silently when running under /bin/dash as " test -a " is not a supported option.
As the "test for seekfile" now fails (even though the file exists), a new seekfile is created with " 0 "
This caused the entire audit.log to be re-read on each iteration

The work-around is to change the first line of the script from #!/bin/sh to #!/bin/bash to ensure bash shell execution.
A defect ticket has been raised with development to rectify this issue (TAG-9906)

Get Updates on the Splunk Community!

Splunk Enterprise Security(ES) 7.3 is approaching the end of support. Get ready for ...

Hi friends!    At Splunk, your product success is our top priority. With Enterprise Security (ES), we're here ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk, and empower your SOC to reach new heights! Duration: 1 hour  Prepare to ...

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...