topic Discard long messages? in Getting Data In

Discard long messages?

BogeyMan — Tue, 25 Feb 2025 19:50:22 GMT

I have an errant application that is sending too much data to my Splunk Enterprise instance.

This is causing licensing overage(s) & warnings.

Until I can fix all the occurrences of this application, I need to configure Splunk to just drop these oversized entries.

I don't want to reject/truncate all messages, just anything over say, 512k.

My understanding is I can do with updates to transform.conf & props.conf?

Here's my transforms.conf:

[drop_unwanted_logs] REGEX = (DEBUG|healthcheck|keepalive) # Drop logs containing these terms DEST_KEY = queue FORMAT = nullQueue [drop_large_events] REGEX = ^.{524288,} # Matches any log >= 512 KB DEST_KEY = queue FORMAT = nullQueue

Ideally, I want this to focus on two of my HEC's, so I updated props.conf:

[source::http:event collector 1] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [source::http:event collector 2] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [sourcetype::http:event collector 1] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [sourcetype::http:event collector 2] TRANSFORMS-null=drop_large_events TRUNCATE = 524288

Am I heading in the right direction? Or, will the following apply to all HEC's?

[sourcetype::httpevent] TRANSFORMS-null=drop_large_events TRUNCATE = 524288

Re: Discard long messages?

livehybrid — Tue, 25 Feb 2025 21:03:29 GMT

Hi @BogeyMan

I guess the main question is, do you want to drop data > 512k, or just truncate it?

If you want to truncate then your TRUNCATE = <n> values should work to truncate to 512k.

Your logic for drop_unwanted_logs also looks good.

I know it might be pseudo-code, for the props.conf, you dont need to specify sourcetype::<yourSourcetype>, its just [<yourSourcetype>]

[source::http:event collector 1] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [source::http:event collector 2] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [sourcetype1] TRANSFORMS-null=drop_large_events TRUNCATE = 524288 [sourcetype2] TRANSFORMS-null=drop_unwanted_logs TRUNCATE = 524288

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

Re: Discard long messages?

BogeyMan — Tue, 25 Feb 2025 22:31:13 GMT

Right now, just looking to drop/discard data > 512k.

If I can get this working, we may refine.

Now, when you refer to "sourcetype", is that "httpevent" (to refer to all defined HECs), or, is that the name of the defined event collector (in my example "event collector 1"?