topic Re: Truncate only certain events from a source in Getting Data In

Truncate only certain events from a source?

maxwee — Wed, 11 Jan 2023 16:11:18 GMT

Good day experts, to manage the ingestion volume, I need apply truncation to a source that sends pretty high volume of data. However, we do not wish to truncate all events from this source, only certain events which are less critical. I tried to override sourcetype in transforms.conf with a regex that matches less critical events and applying truncate in props.conf to the custom sourcetype but failed. Only then I recalled that the transforms and props apply at indexing time. The transforms worked as expected to change the sourcetype but it did not truncate.

Does anyone faced similar use case or can share a way to manage this? Thank you in advance.

Re: Truncate only certain events from a source

gcusello — Tue, 10 Jan 2023 07:44:50 GMT

Hi @maxwee,

you have to find a regex to identify less important events to truncate and then use SEDCMD command in props.conf for that sourcetype.

For more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/Propsconf

Ciao.

Giuseppe

Re: Truncate only certain events from a source

PickleRick — Tue, 10 Jan 2023 12:04:15 GMT

It's not about whether the transforms are applied on index time.

It's about the fact that even though you might overwrite sourcetype, the event's index-time processing will still follow the original sourcetype path, not the new one (unless you use CLONE_SOURCETYPE and discard the original event).

But it's probably easier to use SEDCMD or INGEST_EVAL to selectively "cut" some events.

Re: Truncate only certain events from a source

maxwee — Wed, 11 Jan 2023 08:20:41 GMT

thanks for the advice!