Deployment Architecture

Why is Soucetype Linux_audit generating huge amounts of useless data?

tbalouch
Path Finder

Hi Guys,

I'm troubleshooting excessive license usage for my Splunk cluster and it seems sourcetype = linux_audit is generating a huge amount of data and causing me to go over my license. How would I be able to disable these types of alerts on my splunk instance? Can I do this from the *Nix app or would it be better to specify in the /opt/splunk/etc/system/local/inputs.conf file of the server that is generating the huge amount of traffic? This server also happens to be the search head.

Labels (1)
Tags (3)
0 Karma
1 Solution

doksu
SplunkTrust
SplunkTrust

To stop ingesting linux_audit events, remove (or disable) the rlog.sh scripted stanza in the Splunk_TA_nix app's local inputs.conf being deployed to your universal forwarders. If it's only a problem on one host, then yes you could duplicate Splunk_TA_nix's rlog.sh stanza with the argument 'disabled = true' in $SPLUNK_HOME/etc/system/local/inputs.conf on that particular host.

However, it's probably best to understand why so many linux_audit events are being generated because they're likely indicating a problem and so I wouldn't recommend simply removing them from Splunk. The rlog.sh script is used to resolve UIDs in auditd events to usernames, however as with any scripted input, it introduces more moving parts. I would recommend ingesting the auditd events with a simple monitor stanza on /var/log/audit/audit.log, then use the Linux Auditd app in Splunk (https://splunkbase.splunk.com/app/2642/) to figure out what's going on (it takes care of the UID resolution at search time - which is best practice).

View solution in original post

doksu
SplunkTrust
SplunkTrust

To stop ingesting linux_audit events, remove (or disable) the rlog.sh scripted stanza in the Splunk_TA_nix app's local inputs.conf being deployed to your universal forwarders. If it's only a problem on one host, then yes you could duplicate Splunk_TA_nix's rlog.sh stanza with the argument 'disabled = true' in $SPLUNK_HOME/etc/system/local/inputs.conf on that particular host.

However, it's probably best to understand why so many linux_audit events are being generated because they're likely indicating a problem and so I wouldn't recommend simply removing them from Splunk. The rlog.sh script is used to resolve UIDs in auditd events to usernames, however as with any scripted input, it introduces more moving parts. I would recommend ingesting the auditd events with a simple monitor stanza on /var/log/audit/audit.log, then use the Linux Auditd app in Splunk (https://splunkbase.splunk.com/app/2642/) to figure out what's going on (it takes care of the UID resolution at search time - which is best practice).

tbalouch
Path Finder

Thanks yes I think i disabled it, however I can't stop the linux_audit attempts unless i disable the auditd service. This is very strange.

0 Karma

lukejadamec
Super Champion

I had similar trouble with the app. Out of the box it collects gobs of data, and in my case mostly duplicate data. You should check to see if it is duplicate data, and edit you inputs.conf log file monitoring section to exclude logs you don't want. I also turned of the rslog script because it creates a duplicate of the audit log.

0 Karma

ramitgoyal
New Member

Hello,

I have same kind of issue in the environment.. could you please elaborate in detail on how to identify which logs are useful and which can be omitted from using the splunk license.

We have created a custom index that ingests the auditd logs from all the splunk enterprise instances only which includes all HFs, SHs, and Indexer components.

 

We had disabled the inputs as a workaround as it was breaching our license capacity.

 

Regards

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...