Getting Data In

filter data at indexer from forwarder-- not working

jgauthier
Contributor

I have set up a few heavy forwarders. I did this to filter data, and learn how. Some of these are on a WAN and will remain heavy. However, on my LAN, I'd rather deploy light.

So, I've attempted to move my working configuration to my indexer. However, my indexer is not filtering the data. As a test, I'm just trying to throw everything away.

Currently, I am forwarding only security event log data.

So, on my indexer, I've modified local/props.conf to contain this: (which is a copy from the default)

[source::WinEventLog:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-translate=s/Account EDITED/Account BOOYA/
TRANSFORMS-set = redirect

My local/transforms.conf looks like:

[redirect]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

What I've found is that the SEDCMD is not being executed (It works on the heavies) and the redirect transform is not discarding everything with a source of [source::WinEventLog:Security]

This is all correct, isn't it?

I have tried using just [WinEventLog:Security] as the source, along with attempting to use [host::MACHINE1]

But nothing seems to work.

Thanks.

Tags (2)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

View solution in original post

jgauthier
Contributor

I am still striking out. I set up a light forwarder to test sending squid logs. I set the LWF config to be simple:

[monitor::///var/log/squid/access.log]
followTail=1
host=Squid
sourcetype=squid

On the indexer: props.conf

[squid]
TRANSFORMS-set=setnull,keep

transforms.conf

[keep]
REGEX=(?=)
DEST_KEY=queue
FORMAT=indexQueue

[setnull]
REGEX=TCP_DENIED
DEST_KEY=queue
FORMAT=nullQueue

items with TCP_DENIED are still showing up.

0 Karma

jgauthier
Contributor

(but it works now)

0 Karma

jgauthier
Contributor

I really don't understand what I did differently. I just removed the [keep] stanza...

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

That is correct. The key is to remember to do your props/transforms where parsing occurs. After parsing occurs, the data is cooked and anything further you've specified just won't occur.

0 Karma

jgauthier
Contributor

Okay. So I can definitely do this with a LWF, but not a heavy?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!