All Apps and Add-ons

Metrics collecting bash scripts generating triggering too many auditd alerts

New Member

Hi,

We're using a simplified version of the SplunkTAnix app (basically just a subset of the bash scripts) to collect system metrics such as CPU memory disk usages;

It works well, but the problem is that these bash commands are constantly triggering our auditd rules on the end point (which is configured to monitor for execve syscalls), at an extremely high rate. E.g. every 30 seconds, one of these bash scripts executes, which triggers ~ 20 audit events, resulting in 100+ audit records, just from the sed, awk, tee, wc etc. commands within the script, which get fed back into splunk via the other auditd app we have (the amount of data generated from this is huge - larger than normal system activities in general).

One solution we're considering is to filter out audit records with auid=-1 from the audit rules side (with the tradeoff of losing some visibility);
Another approach is to rewrite all these bash scripts in python or some compiled languages so that it doesn't trigger that many execve syscalls;
Third option is to filter out the specific commands generated by these scripts;

Does anyone else have a similar problem? What's the best way to deal with this kind of issues? Is there another good app that serves a similar purpose but does not use bash (using python / C etc. instead)?

Any info is really appreciated!

0 Karma

Engager

Hello @tzhang_gr ,
If I correctly understand your problem, it seems that the scripts in the TA for Unix and LInux leave too much of a footprint on the system.
If that is the case, might it be possible for you to use collectd to get the metrics you seek?
https://collectd.org/

You could then visualize the data with the Splunk App for Infrastructure.
https://splunkbase.splunk.com/app/3975/

Regards,
Hans

0 Karma

New Member

Thank you very much! I'll take a look into this approach.

0 Karma