topic Re: Same sourcetype, but different transforms per Host in Getting Data In

Same sourcetype, but different transforms per Host

adylent — Wed, 06 Nov 2013 20:51:56 GMT

I have an issue where we have a sourcetype that we want to remove a transform (on the indexer) that drops some data (but only for a specific host wildcard pattern). I've been tasked with keeping the same sourcetype name, but not applying the transforms to outputs from a series of hosts.

We already use a props [Host: ] stanza for timezones. Is there a simple way to keep the same sourcetype, but force inputs from a particular host pattern to follow a different set of transforms? Preferrably a host (and sourcetype) specific transform.

Thanks

Re: Same sourcetype, but different transforms per Host

lguinn2 — Wed, 06 Nov 2013 21:38:16 GMT

It can be hard to perform both a host and sourcetype-specific transformation. It depends on what you want to be done.

In props.conf, select the sourcetype. First, apply the usual transformation, then apply the exception case

[mysourcetype]
TRANSFORMS-t1=firstTransform,secondTransform

In transforms.conf, send the data as needed. In the first transform, do whatever you are doing now. I just put in the nullQueue to have a complete example. The second transform selects the data based on the host name, not based on the event content.

[firstTransform]
REGEX = what_to_delete_based_on_raw_data
DEST_KEY = queue
FORMAT = nullQueue

[secondTransform]
SOURCE_KEY = MetaData:Host
REGEX = pattern_to_match_host_name
DEST_KEY = queue
FORMAT = indexQueue

You can read more about this at Filter Event Data and Send to Queues

Please test this carefully and be sure to use regular expressions, not just "wildcards." I can't test it, so I might have made mistakes...

Re: Same sourcetype, but different transforms per Host

adylent — Thu, 07 Nov 2013 21:36:39 GMT

This is a really good answer. Thank you.

The props/transforms I'm working on have a mix of REPORT / EXTRACTS. I endedup adding the rule at the end of the REPORT section and this looks to have done the trick in our development instance.