Getting Data In
Highlighted

filter data at indexer from forwarder-- not working

Contributor

I have set up a few heavy forwarders. I did this to filter data, and learn how. Some of these are on a WAN and will remain heavy. However, on my LAN, I'd rather deploy light.

So, I've attempted to move my working configuration to my indexer. However, my indexer is not filtering the data. As a test, I'm just trying to throw everything away.

Currently, I am forwarding only security event log data.

So, on my indexer, I've modified local/props.conf to contain this: (which is a copy from the default)

[source::WinEventLog:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-translate=s/Account EDITED/Account BOOYA/
TRANSFORMS-set = redirect

My local/transforms.conf looks like:

[redirect]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

What I've found is that the SEDCMD is not being executed (It works on the heavies) and the redirect transform is not discarding everything with a source of [source::WinEventLog:Security]

This is all correct, isn't it?

I have tried using just [WinEventLog:Security] as the source, along with attempting to use [host::MACHINE1]

But nothing seems to work.

Thanks.

Tags (2)
Highlighted

Re: filter data at indexer from forwarder-- not working

Influencer

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

View solution in original post

Highlighted

Re: filter data at indexer from forwarder-- not working

Contributor

Okay. So I can definitely do this with a LWF, but not a heavy?

0 Karma
Highlighted

Re: filter data at indexer from forwarder-- not working

Influencer

That is correct. The key is to remember to do your props/transforms where parsing occurs. After parsing occurs, the data is cooked and anything further you've specified just won't occur.

0 Karma
Highlighted

Re: filter data at indexer from forwarder-- not working

Contributor

I am still striking out. I set up a light forwarder to test sending squid logs. I set the LWF config to be simple:

[monitor::///var/log/squid/access.log]
followTail=1
host=Squid
sourcetype=squid

On the indexer: props.conf

[squid]
TRANSFORMS-set=setnull,keep

transforms.conf

[keep]
REGEX=(?=)
DEST_KEY=queue
FORMAT=indexQueue

[setnull]
REGEX=TCP_DENIED
DEST_KEY=queue
FORMAT=nullQueue

items with TCP_DENIED are still showing up.

0 Karma
Highlighted

Re: filter data at indexer from forwarder-- not working

Contributor

I really don't understand what I did differently. I just removed the [keep] stanza...

0 Karma
Highlighted

Re: filter data at indexer from forwarder-- not working

Contributor

(but it works now)

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.