topic transforms.conf: Need to ignore specific events. in Getting Data In

transforms.conf: Need to ignore specific events.

Egyas — Mon, 04 Mar 2024 22:01:37 GMT

I have a simply Splunk set-up. about 120 or so Linux servers (that are all basically appliances) w/ universal forwarder installed, and a single Linux server running Splunk Enterprise acting as the indexer, search head, etc.

The problem I have is that the forwarders must feed the server's audit log into Splunk. That feed is actually working fine, but it's flooding the server, and causing me to go over my license limit.

Specifically, the appliance app has an event in cron that runs very often, and it's flooding the audit log with file access, file mod, etc events, which is ballooning the amount of data I send to Splunk Enterprise. Data that IO simply do not need.

What I want to do is filter out these specific events, but ONLY for this specific user. I believe this can be done using transforms.conf and props.conf on the indexer, but I'm having trouble getting the syntax and fields right.

Can anyone assist with this?

Here's the data I need to remove...

sourcetype=auditd acct=appuser exe=/usr/sbin/crond exe=/usr/bin/crontab

So basically ANY events in the audit log for user "appuser" that reference either "/usr/bin/crontab" or "usr/bin/crontab" need to be dropped.

Here are 2 examples of the events I want to drop.

type=USER_END msg=audit(03/04/2024 15:58:02.701:5726) : pid=26919 uid=root auid=appuser ses=184 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct=appuser exe=/usr/sbin/crond hostname=? addr=? terminal=cron res=success' type=USER_ACCT msg=audit(03/04/2024 15:58:02.488:5723) : pid=26947 uid=appuser auid=appuser ses=184 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct=appuser exe=/usr/bin/crontab hostname=? addr=? terminal=cron res=success'

Can this be done?

Re: transforms.conf: Need to ignore specific events.

kiran_panchavat — Tue, 05 Mar 2024 07:31:42 GMT

@Egyas Hello, You can drop the events using props.conf and transforms.conf. So, first thing you have to match the events which one you want to drop using regex. Let's say if you want to drop the event called "acct=appuser", write the regex for that and apply the props.conf and transforms.conf and send those data to the null queue.

Example:

props.conf

[source::xxxxx]
TRANSFORMS-set=setnull

Transforms.conf

[setnull]
REGEX = <your regex> i.e., acct=appuser
DEST_KEY = queue
FORMAT = nullQueue

https://docs.splunk.com/Documentation/Splunk/9.2.0/Admin/Propsconf#props.conf.example

https://docs.splunk.com/Documentation/Splunk/9.2.0/Admin/Transformsconf

* NOTE: Keys are case-sensitive. Use the following keys exactly as they
        appear.

queue : Specify which queue to send the event to (can be nullQueue, indexQueue).
        * indexQueue is the usual destination for events going through the
          transform-handling processor.
        * nullQueue is a destination which causes the events to be
          dropped entirely.
_raw  : The raw text of the event.
_meta : A space-separated list of metadata for an event.
_time : The timestamp of the event, in seconds since 1/1/1970 UTC.

TRANSFORMS-<class> = <transform_stanza_name>, <transform_stanza_name2>,...
* Used for creating indexed fields (index-time field extractions).
* <class> is a unique literal string that identifies the namespace of the
  field you're extracting.
  **Note:** <class> values do not have to follow field name syntax
  restrictions. You can use characters other than a-z, A-Z, and 0-9, and
  spaces are allowed. <class> values are not subject to key cleaning.
* <transform_stanza_name> is the name of your stanza from transforms.conf.
* Use a comma-separated list to apply multiple transform stanzas to a single
  TRANSFORMS extraction. Splunk software applies them in the list order. For
  example, this sequence ensures that the [yellow] transform stanza gets
  applied first, then [blue], and then [red]:
        [source::color_logs]
        TRANSFORMS-colorchange = yellow, blue, red
* See the RULESET-<class> setting for additional index-time transformation options.

Re: transforms.conf: Need to ignore specific events.

PickleRick — Tue, 05 Mar 2024 08:30:08 GMT

Luckily, with auditd logs the order of the fields should not change so you can match the events to

acct=appuser.*exe=/usr/(sbin/crond|bin/crontab)

and just filter out (send to nullQueue) events matching this regex.

Re: transforms.conf: Need to ignore specific events.

Egyas — Tue, 05 Mar 2024 16:56:24 GMT

@PickleRick & @kiran_panchavat , thank you guys so much for the assist. I really appreciate it. I'll give it a test and see if it works for me. Thanks agaion!

Re: transforms.conf: Need to ignore specific events.

Egyas — Tue, 05 Mar 2024 18:56:45 GMT

I just tested this and it works perfectly. I tweaked a few names and combined the file contents from @kiran_panchavat with the regex from @PickleRick and I'm good to go. Thanks guys!

props.conf

[source::auditd] TRANSFORMS-set=setnull

transforms.conf

[setnull] REGEX = acct=appuser.*exe=/usr/(sbin/crond|bin/crontab) DEST_KEY = queue FORMAT = nullQueue