I have set up a few heavy forwarders. I did this to filter data, and learn how. Some of these are on a WAN and will remain heavy. However, on my LAN, I'd rather deploy light.
So, I've attempted to move my working configuration to my indexer. However, my indexer is not filtering the data. As a test, I'm just trying to throw everything away.
Currently, I am forwarding only security event log data.
So, on my indexer, I've modified local/props.conf to contain this: (which is a copy from the default)
[source::WinEventLog:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-translate=s/Account EDITED/Account BOOYA/
TRANSFORMS-set = redirect
My local/transforms.conf looks like:
[redirect]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue
What I've found is that the SEDCMD is not being executed (It works on the heavies) and the redirect transform is not discarding everything with a source of [source::WinEventLog:Security]
This is all correct, isn't it?
I have tried using just [WinEventLog:Security]
as the source, along with attempting to use [host::MACHINE1]
But nothing seems to work.
Thanks.
That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.
I am still striking out. I set up a light forwarder to test sending squid logs. I set the LWF config to be simple:
[monitor::///var/log/squid/access.log]
followTail=1
host=Squid
sourcetype=squid
On the indexer: props.conf
[squid]
TRANSFORMS-set=setnull,keep
transforms.conf
[keep]
REGEX=(?=)
DEST_KEY=queue
FORMAT=indexQueue
[setnull]
REGEX=TCP_DENIED
DEST_KEY=queue
FORMAT=nullQueue
items with TCP_DENIED are still showing up.
(but it works now)
I really don't understand what I did differently. I just removed the [keep] stanza...
That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.
That is correct. The key is to remember to do your props/transforms where parsing occurs. After parsing occurs, the data is cooked and anything further you've specified just won't occur.
Okay. So I can definitely do this with a LWF, but not a heavy?