Discard long messages?

[BogeyMan]

I have an errant application that is sending too much data to my Splunk Enterprise instance.

This is causing licensing overage(s) & warnings.

Until I can fix all the occurrences of this application, I need to configure Splunk to just drop these oversized entries.

I don't want to reject/truncate all messages, just anything over say, 512k.

My understanding is I can do with updates to transform.conf & props.conf?

Here's my transforms.conf:

[drop_unwanted_logs]
REGEX = (DEBUG|healthcheck|keepalive)  # Drop logs containing these terms
DEST_KEY = queue
FORMAT = nullQueue

[drop_large_events]
REGEX = ^.{524288,}  # Matches any log >= 512 KB
DEST_KEY = queue
FORMAT = nullQueue

Ideally, I want this to focus on two of my HEC's, so I updated props.conf:

[source::http:event collector 1]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[source::http:event collector 2]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[sourcetype::http:event collector 1]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[sourcetype::http:event collector 2]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

Am I heading in the right direction? Or, will the following apply to all HEC's?

[sourcetype::httpevent]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[livehybrid]

Hi @BogeyMan 

I guess the main question is, do you want to drop data > 512k, or just truncate it?

If you want to truncate then your TRUNCATE = <n> values should work to truncate to 512k.

Your logic for drop_unwanted_logs also looks good. 

I know it might be pseudo-code, for the props.conf, you dont need to specify sourcetype::<yourSourcetype>, its just [<yourSourcetype>]

[source::http:event collector 1]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[source::http:event collector 2]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[sourcetype1]
TRANSFORMS-null=drop_large_events
TRUNCATE = 524288

[sourcetype2]
TRANSFORMS-null=drop_unwanted_logs
TRUNCATE = 524288

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[BogeyMan]

Right now, just looking to drop/discard data > 512k.

If I can get this working, we may refine.

Now, when you refer to "sourcetype", is that "httpevent" (to refer to all defined HECs), or, is that the name of the defined event collector (in my example "event collector 1"?