All Apps and Add-ons

Why does Splunk_TA_nix rlog.sh cause huge amount of duplicate audit events on Ubuntu?

dshakespeare_sp
Splunk Employee
Splunk Employee

Customer reported that after enabling rlog.sh on Ubuntu Universal Forwarder they were getting 4 x the number of events they were expecting. From a 500,000 line audit.log, they received over 2 million events in about 10 minutes. Further investigation showed that events were being duplicated by a factor of 4.

dshakespeare_sp
Splunk Employee
Splunk Employee

Audit.log is read by a scripted input ( rlog.sh ) which runs every 60 secs
The script keeps is own seek_pointer ( unix_audit_seekfile ) in $SPLUNK_HOME/var/run/splunk
The problem is caused by the fact that the script runs under #!/bin/sh
Part of the code tests for the existence of unix_audit_seekfile with [ -a $SEEK_FILE ]
If the file does not exist, we create a new pointer " 0 "

On most systems /bin/sh is a link to /bin/bash BUT on Ubuntu it is a link to /bin/dash
[ -a $SEEK_FILE ] will fail silently when running under /bin/dash as " test -a " is not a supported option.
As the "test for seekfile" now fails (even though the file exists), a new seekfile is created with " 0 "
This caused the entire audit.log to be re-read on each iteration

The work-around is to change the first line of the script from #!/bin/sh to #!/bin/bash to ensure bash shell execution.
A defect ticket has been raised with development to rectify this issue (TAG-9906)

Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...