Getting Data In

filter data at indexer from forwarder-- not working

jgauthier
Contributor

I have set up a few heavy forwarders. I did this to filter data, and learn how. Some of these are on a WAN and will remain heavy. However, on my LAN, I'd rather deploy light.

So, I've attempted to move my working configuration to my indexer. However, my indexer is not filtering the data. As a test, I'm just trying to throw everything away.

Currently, I am forwarding only security event log data.

So, on my indexer, I've modified local/props.conf to contain this: (which is a copy from the default)

[source::WinEventLog:Security]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none
TRANSFORMS-FIELDS = strip-winevt-linebreaker
SEDCMD-translate=s/Account EDITED/Account BOOYA/
TRANSFORMS-set = redirect

My local/transforms.conf looks like:

[redirect]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue

What I've found is that the SEDCMD is not being executed (It works on the heavies) and the redirect transform is not discarding everything with a source of [source::WinEventLog:Security]

This is all correct, isn't it?

I have tried using just [WinEventLog:Security] as the source, along with attempting to use [host::MACHINE1]

But nothing seems to work.

Thanks.

Tags (2)
1 Solution

jbsplunk
Splunk Employee
Splunk Employee

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

View solution in original post

jgauthier
Contributor

I am still striking out. I set up a light forwarder to test sending squid logs. I set the LWF config to be simple:

[monitor::///var/log/squid/access.log]
followTail=1
host=Squid
sourcetype=squid

On the indexer: props.conf

[squid]
TRANSFORMS-set=setnull,keep

transforms.conf

[keep]
REGEX=(?=)
DEST_KEY=queue
FORMAT=indexQueue

[setnull]
REGEX=TCP_DENIED
DEST_KEY=queue
FORMAT=nullQueue

items with TCP_DENIED are still showing up.

0 Karma

jgauthier
Contributor

(but it works now)

0 Karma

jgauthier
Contributor

I really don't understand what I did differently. I just removed the [keep] stanza...

0 Karma

jbsplunk
Splunk Employee
Splunk Employee

That isn't going to work, because once the data arrives on the indexer, it is going to be cooked. Once the indexer sees cooked data, it won't do anything else with it. The changes need to happen where the data is parsed, which in the situation you are describing, is on the full Forwarder. If you were using an LWF, this would work on the indexer.

jbsplunk
Splunk Employee
Splunk Employee

That is correct. The key is to remember to do your props/transforms where parsing occurs. After parsing occurs, the data is cooked and anything further you've specified just won't occur.

0 Karma

jgauthier
Contributor

Okay. So I can definitely do this with a LWF, but not a heavy?

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...