Why is Soucetype Linux_audit generating huge amounts of useless data?

[tbalouch]

Hi Guys,

I'm troubleshooting excessive license usage for my Splunk cluster and it seems sourcetype = linux_audit is generating a huge amount of data and causing me to go over my license. How would I be able to disable these types of alerts on my splunk instance? Can I do this from the *Nix app or would it be better to specify in the /opt/splunk/etc/system/local/inputs.conf file of the server that is generating the huge amount of traffic? This server also happens to be the search head.

[doksu · Accepted solution]

To stop ingesting linux_audit events, remove (or disable) the rlog.sh scripted stanza in the Splunk_TA_nix app's local inputs.conf being deployed to your universal forwarders. If it's only a problem on one host, then yes you could duplicate Splunk_TA_nix's rlog.sh stanza with the argument 'disabled = true' in $SPLUNK_HOME/etc/system/local/inputs.conf on that particular host.

However, it's probably best to understand why so many linux_audit events are being generated because they're likely indicating a problem and so I wouldn't recommend simply removing them from Splunk. The rlog.sh script is used to resolve UIDs in auditd events to usernames, however as with any scripted input, it introduces more moving parts. I would recommend ingesting the auditd events with a simple monitor stanza on /var/log/audit/audit.log, then use the Linux Auditd app in Splunk (https://splunkbase.splunk.com/app/2642/) to figure out what's going on (it takes care of the UID resolution at search time - which is best practice).

[tbalouch]

Thanks yes I think i disabled it, however I can't stop the linux_audit attempts unless i disable the auditd service. This is very strange.

[lukejadamec]

I had similar trouble with the app. Out of the box it collects gobs of data, and in my case mostly duplicate data. You should check to see if it is duplicate data, and edit you inputs.conf log file monitoring section to exclude logs you don't want. I also turned of the rslog script because it creates a duplicate of the audit log.

[ramitgoyal]

Hello,

I have same kind of issue in the environment.. could you please elaborate in detail on how to identify which logs are useful and which can be omitted from using the splunk license.

We have created a custom index that ingests the auditd logs from all the splunk enterprise instances only which includes all HFs, SHs, and Indexer components.

We had disabled the inputs as a workaround as it was breaching our license capacity.

Regards