topic How to filter out specific events sent via Universal Forwarder? in Getting Data In

How to filter out specific events sent via Universal Forwarder?

rkymtnhigh — Sun, 07 Jun 2020 00:25:38 GMT

I have one indexer that is receiving events from a remote Windows host via the Universal Forwarder.

I am trying to filter out events that contain the string 'empty logger' in the log file D:\Logs\Test\testlog5_29_20.log file on the remote server.

I have attempted to use the props.conf and the transforms.conf files on the indexer to send the events matching the regex to nullqueue, but the events in question are still making it.

I am suspecting that the source stanza in the props.conf file isn't correct, as I am specifying a directory that only exists on the remote Windows hosts.

Am I correct in that assumption?

Re: How to filter out specific events sent via Universal Forwarder?

richgalloway — Fri, 29 May 2020 20:07:09 GMT

Please share a sample event (sanitized) and your current props and transforms for the sourcetype.

Re: How to filter out specific events sent via Universal Forwarder?

mchristopherson — Fri, 29 May 2020 20:49:14 GMT

We are doing this in the following fashion - but we would need to see how you have your configs formatted:

props.conf

[sourcetype:to:modify]
TRANSFORMS-null = StanzaNameInTransforms

This is simply the name we are giving it. It must start with TRANSFORMS but you can use -"name" to have multiple TRANSFORMS on one sourcetype.

transforms.conf

[StanzaNameInTransforms]
REGEX = 
DEST_KEY= queue
FORMAT = nullQueue

Your REGEX can be a partial portion of a line. I would play around with that bit but in one of our examples, we simply have a string that shows up in our examples we want dropped. From your example it should be:

[StanzaNameInTransforms]
REGEX = empty logger
DEST_KEY= queue
FORMAT = nullQueue

Re: How to filter out specific events sent via Universal Forwarder?

rkymtnhigh — Mon, 01 Jun 2020 16:21:45 GMT

I think my issues lies with my props.conf source..
I am currently using:

[source::d:\logs\...\*.log]
TRANSFORMS-null = setnull

I tried using the sourcetype that shows up when I search these events "SampleSourcetype2"

Here is how I have it set:

[sourcetype::SampleSourcetype2]
TRANSFORMS-null = setnull

Then in transforms.conf

[setnull]
REGEX = empty logger
DEST_KEY = queue
FORMAT = nullQueue

The events with "empty logger" are still being indexed however.

Re: How to filter out specific events sent via Universal Forwarder?

richgalloway — Mon, 01 Jun 2020 16:46:11 GMT

That syntax looks OK (well not so much. See my later comment). What does the [setnull] stanza in your transforms.conf file look like?

Re: How to filter out specific events sent via Universal Forwarder?

rkymtnhigh — Mon, 01 Jun 2020 16:47:20 GMT

 [setnull]
 REGEX = empty logger
 DEST_KEY = queue
 FORMAT = nullQueue

Re: How to filter out specific events sent via Universal Forwarder?

rkymtnhigh — Mon, 01 Jun 2020 19:07:41 GMT

Resolved the issue using [source::....log] stanza in props.conf.
Thanks everyone for your help.

Re: How to filter out specific events sent via Universal Forwarder?

richgalloway — Mon, 01 Jun 2020 20:16:34 GMT

BTW, [sourcetype::SampleSourcetype2] is not supported in props.conf. Use [SampleSourcetype2].
Also, the backslashes must be escaped. source::d:\\logs\\...\\*.log