topic Re: Filtering events from forwarder at indexer in Getting Data In

Filtering events from forwarder at indexer

twinspop — Wed, 10 Nov 2010 07:45:46 GMT

I'm trying to filter noisy events that have recently pushed us over license usage. The events come from a lightweight forwarder that I don't have access to at the moment. I'd like to filter at the indexer. This is not working:

props.conf:

[source::/Logs/Noisy.log]
TRANSFORMS-set = setnull

transforms.conf:

[setnull]
REGEX = host07
DEST_KEY = queue
FORMAT = nullQueue

Where "host07" is present in all the logs I want to dump, and the source matches that displayed in search results on the indexer. I've also tried listing the sourcetype as shown in search results.

Gotta be something simple... or maybe LWF traffic needs to be filtered on the LWF, not the indexer?

Thanks, Jon

Re: Filtering events from forwarder at indexer

dwaddle — Wed, 10 Nov 2010 11:18:08 GMT

Lightweight Forwarder requires filtering to be done at the indexer -- it cannot do it itself.

The answer http://splunk-base.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk has some good examples of using nullQueue.

At a quick glance, your example looks fundamentally correct. Does "host07" appear in the _raw for the events in question, or is it in another metadata field?

Re: Filtering events from forwarder at indexer

twinspop — Wed, 10 Nov 2010 12:18:55 GMT

Yes, the string is in _raw.

Re: Filtering events from forwarder at indexer

the_wolverine — Wed, 10 Nov 2010 14:54:17 GMT

Are you sure its a LWF? If not, the data is already cooked and cannot be manipulated at the indexer.

Re: Filtering events from forwarder at indexer

twinspop — Wed, 10 Nov 2010 22:44:12 GMT

Is there any way to tell from the indexer's side? I did not do the install, just gave a list of command line instructions. It's possible the server admin didn't follow them.