Truncate only certain events from a source?

[maxwee]

Good day experts, to manage the ingestion volume, I need apply truncation to a source that sends pretty high volume of data. However, we do not wish to truncate all events from this source, only certain events which are less critical. I tried to override sourcetype in transforms.conf with a regex that matches less critical events and applying truncate in props.conf to the custom sourcetype but failed. Only then I recalled that the transforms and props apply at indexing time. The transforms worked as expected to change the sourcetype but it did not truncate. 

Does anyone faced similar use case or can share a way to manage this? Thank you in advance.

[maxwee]

thanks for the advice!

[PickleRick · Accepted solution]

It's not about whether the transforms are applied on index time.

It's about the fact that even though you might overwrite sourcetype, the event's index-time processing will still follow the original sourcetype path, not the new one (unless you use CLONE_SOURCETYPE and discard the original event).

But it's probably easier to use SEDCMD or INGEST_EVAL to selectively "cut" some events.

[gcusello]

Hi @maxwee,

you have to find a regex to identify less important events to truncate and then use SEDCMD command in props.conf for that sourcetype.

For more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.3/Admin/Propsconf

Ciao.

Giuseppe