topic filter data at indexer from forwarder-- not working in Getting Data In

filter data at indexer from forwarder-- not working

jgauthier — Sat, 26 Mar 2011 01:57:34 GMT

I have set up a few heavy forwarders. I did this to filter data, and learn how. Some of these are on a WAN and will remain heavy. However, on my LAN, I'd rather deploy light.

So, I've attempted to move my working configuration to my indexer. However, my indexer is not filtering the data. As a test, I'm just trying to throw everything away.

Currently, I am forwarding only security event log data.

So, on my indexer, I've modified local/props.conf to contain this: (which is a copy from the default)

[source::WinEventLog:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-translate=s/Account EDITED/Account BOOYA/
TRANSFORMS-set = redirect

My local/transforms.conf looks like:

[redirect]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

What I've found is that the SEDCMD is not being executed (It works on the heavies) and the redirect transform is not discarding everything with a source of [source::WinEventLog:Security]

This is all correct, isn't it?

I have tried using just [WinEventLog:Security] as the source, along with attempting to use [host::MACHINE1]

But nothing seems to work.

Thanks.

Re: filter data at indexer from forwarder-- not working

jbsplunk — Sat, 26 Mar 2011 02:10:00 GMT

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

Re: filter data at indexer from forwarder-- not working

jgauthier — Sat, 26 Mar 2011 02:12:33 GMT

Okay. So I can definitely do this with a LWF, but not a heavy?

Re: filter data at indexer from forwarder-- not working

jbsplunk — Sat, 26 Mar 2011 02:35:39 GMT

That is correct. The key is to remember to do your props/transforms where parsing occurs. After parsing occurs, the data is cooked and anything further you've specified just won't occur.

Re: filter data at indexer from forwarder-- not working

jgauthier — Sat, 26 Mar 2011 03:16:00 GMT

I am still striking out. I set up a light forwarder to test sending squid logs. I set the LWF config to be simple:

[monitor::///var/log/squid/access.log]
followTail=1
host=Squid
sourcetype=squid

On the indexer: props.conf

[squid]
TRANSFORMS-set=setnull,keep

transforms.conf

[keep]
REGEX=(?=)
DEST_KEY=queue
FORMAT=indexQueue

[setnull]
REGEX=TCP_DENIED
DEST_KEY=queue
FORMAT=nullQueue

items with TCP_DENIED are still showing up.

Re: filter data at indexer from forwarder-- not working

jgauthier — Sat, 26 Mar 2011 07:10:49 GMT

I really don't understand what I did differently. I just removed the [keep] stanza...

Re: filter data at indexer from forwarder-- not working

jgauthier — Sat, 26 Mar 2011 07:11:13 GMT

(but it works now)