topic How to configure a Forwarder to filter and send only the events I want? in Getting Data In

How to configure a Forwarder to filter and send only the events I want?

maverick — Wed, 17 Nov 2010 07:09:35 GMT

I have a temporary need to filter and forward ONLY a specific set of events to my indexer.

I see from a couple other answers already posted that I can blacklist to the nullQueue, etc, prior to forwarding.

However, in this case, what I really need to do is whitelist down to a few WinEvent codes (and possibly a few other text patterns) that are required to be indexed, and then only send THOSE events.

Re: How to configure a Forwarder to filter and send only the events I want?

maverick — Wed, 17 Nov 2010 07:10:07 GMT

One way to achieve this goal of whitelisting only events you want from the forwarder (which BTW, defeats the whole purpose of using a forwarder to begin with, IMHO, but thats whole other story in itself), you can setup your default output TCP routing queue to be a nonexistent ip and port in your outputs.conf, therefore, turning off all event forwarding by default. Then you can setup a second TCP output queue to use for forwarding ONLY the events that match you regular expression pattern in transforms.conf file. Finally, you can associate the whitelist matching congif in your props.conf with your sourcetypes, host, or source, per usual setup.

Below is a whitelisting configuration I tested and that works on a heavy forwarder running on Windows.

Please note that in the last line in the config below, you should replace the server value that says "" with the value that matches the ip address of your own Splunk indexing server before restarting.

# props.conf
# --------------
[WinEventLog:System]
TRANSFORMS-set = allowtheseevents

[WinEventLog:Security]
TRANSFORMS-sec = allowtheseevents

# transforms.conf
# ---------------------
[allowtheseevents]
REGEX = (?msi).*?EventCode\=(4624|4648|4672|4778|7035|7036).*
DEST_KEY = _TCP_ROUTING
FORMAT = allowedEventsGroup

# outputs.conf
# ----------------
[tcpout]
defaultGroup=nullGroup
indexAndForward = 0

[tcpout:nullGroup]
server=0.0.0.0:0000

[tcpout:allowedEventsGroup]
server=<your_indexing_ip_here>:9997

Re: How to configure a Forwarder to filter and send only the events I want?

bwooden — Wed, 17 Nov 2010 07:37:43 GMT

The challenge with only forwarding very specific things is that sometimes we don't know what we'll need until we need it. 🙂

If you're using a LWF you could start by forwarding anything that contains the events you want to index. On the indexer you would route all of that to the null queue in one transforms and trump that in the second transforms that will include what you're specifically interested in.

If your needs are very specific, you could also send them along via a scripted input.

Re: How to configure a Forwarder to filter and send only the events I want?

maverick — Wed, 17 Nov 2010 07:51:42 GMT

Totally agree. It can be very dangerous to whitelist events because you will never know what you are missing.

However, if you are required to, then whitelisting from the forward side would cut down on network bandwidth.

Re: How to configure a Forwarder to filter and send only the events I want?

ageld — Tue, 05 Apr 2011 01:13:17 GMT

Question:

If I made configuration changes mentioned by Maverick, how would I send other logs/events to the same indexer. For example, how would I send DHCP logs, WindowsUpdate logs, WMI stuff? Will I have to tweak props.conf, transforms.conf, output.conf for every log?

Re: How to configure a Forwarder to filter and send only the events I want?

ageld — Tue, 05 Apr 2011 02:31:58 GMT

Also, which application should I make the changes to props.conf, transforms.conf, and output.conf? I tried to do it under F:\Program Files\Splunk\etc\apps\SplunkForwarder\, but it did not make a difference. No events appeared on the idexer.