Getting Data In

How to configure a Forwarder to filter and send only the events I want?

maverick
Splunk Employee
Splunk Employee

I have a temporary need to filter and forward ONLY a specific set of events to my indexer.

I see from a couple other answers already posted that I can blacklist to the nullQueue, etc, prior to forwarding.

However, in this case, what I really need to do is whitelist down to a few WinEvent codes (and possibly a few other text patterns) that are required to be indexed, and then only send THOSE events.

ageld
Path Finder

Question:

If I made configuration changes mentioned by Maverick, how would I send other logs/events to the same indexer. For example, how would I send DHCP logs, WindowsUpdate logs, WMI stuff? Will I have to tweak props.conf, transforms.conf, output.conf for every log?

0 Karma

ageld
Path Finder

Also, which application should I make the changes to props.conf, transforms.conf, and output.conf? I tried to do it under F:\Program Files\Splunk\etc\apps\SplunkForwarder\, but it did not make a difference. No events appeared on the idexer.

0 Karma

bwooden
Splunk Employee
Splunk Employee

The challenge with only forwarding very specific things is that sometimes we don't know what we'll need until we need it. 🙂

If you're using a LWF you could start by forwarding anything that contains the events you want to index. On the indexer you would route all of that to the null queue in one transforms and trump that in the second transforms that will include what you're specifically interested in.

If your needs are very specific, you could also send them along via a scripted input.

maverick
Splunk Employee
Splunk Employee

Totally agree. It can be very dangerous to whitelist events because you will never know what you are missing.

However, if you are required to, then whitelisting from the forward side would cut down on network bandwidth.

0 Karma

maverick
Splunk Employee
Splunk Employee

One way to achieve this goal of whitelisting only events you want from the forwarder (which BTW, defeats the whole purpose of using a forwarder to begin with, IMHO, but thats whole other story in itself), you can setup your default output TCP routing queue to be a nonexistent ip and port in your outputs.conf, therefore, turning off all event forwarding by default. Then you can setup a second TCP output queue to use for forwarding ONLY the events that match you regular expression pattern in transforms.conf file. Finally, you can associate the whitelist matching congif in your props.conf with your sourcetypes, host, or source, per usual setup.

Below is a whitelisting configuration I tested and that works on a heavy forwarder running on Windows.

Please note that in the last line in the config below, you should replace the server value that says "" with the value that matches the ip address of your own Splunk indexing server before restarting.

# props.conf
# --------------
[WinEventLog:System]
TRANSFORMS-set = allowtheseevents

[WinEventLog:Security]
TRANSFORMS-sec = allowtheseevents



# transforms.conf
# ---------------------
[allowtheseevents]
REGEX = (?msi).*?EventCode\=(4624|4648|4672|4778|7035|7036).*
DEST_KEY = _TCP_ROUTING
FORMAT = allowedEventsGroup



# outputs.conf
# ----------------
[tcpout]
defaultGroup=nullGroup
indexAndForward = 0

[tcpout:nullGroup]
server=0.0.0.0:0000

[tcpout:allowedEventsGroup]
server=<your_indexing_ip_here>:9997

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...