Getting Data In

How to configure a forwarder to filter and send the specific events I want?

fernandoandre
Communicator

I'm using a set of universal forwarders to send data to a central indexer.

I would like to send events from "WinEventLog:Security" only if, for example, the Event Code is 552 (EventCode=552).

I have read some posts about the same subject, and try some of the suggested solutions. However I haven't been able to make scenarios similar to this one to work, since I still receive all types of events.

My conf files:

inputs.conf

----------------

[WinEventLog:Security]

disabled = 0

index = my_index

start_from = oldest

props.conf

--------------

[WinEventLog:Security]

TRANSFORMS-sec = allowtheseevents

outputs.conf

----------------

[tcpout]

defaultGroup=nullGroup

indexAndForward = 0

[tcpout:nullGroup]

server=0.0.0.0:0000

[tcpout:allowedEventsGroup]

server=(my_server):9997

transforms.conf

---------------------

[allowtheseevents]

REGEX = (?m)^EventCode=552

DEST_KEY = _TCP_ROUTING

FORMAT = allowedEventsGroup

On "transforms.conf" I have also tried something like: "[\w\W]+EventCode\s*=\s*552[\w\W]+"
Can someone help me on this? Thank you.

0 Karma
1 Solution

tgow
Splunk Employee
Splunk Employee

First of all you will not be able to filter on the Universal Forwarder. If you want to filter events on the Windows server then you will need to install a regular/heavy Forwarder. If you want to continue using a UF instead then you will need to modify the config files on the Indexer. Here is a link to information on how to install a Regular/Heavy Forwarder:

http://docs.splunk.com/Documentation/Splunk/4.2.5/Deploy/Deployaforwarder

View solution in original post

BenTan
Path Finder

You can install the Windows TA onto the Universal Forwarder and modify the inputs.conf using whitelist or blacklist depending on the usage. At least thats what I did. Hope it helps!

0 Karma

lqiao
Explorer

Yes, you can use Windows TA or simply in the UF inputs.conf, configure the whitelist or blacklist. The whitelist or blacklist functionality exists only from UF v6 onwards.

The original post was in 2012, probably at that time, the solution was to do the regex on the indexer or heavy forwarder layer.

0 Karma

tobbedahl
New Member

Hi.
I am also looking for a way to only index a few windoes event ID:s, and since I found out that this cannot be done at the universal forwarder I'm looking for a way do this on the indexer.
Fernandoandre, did you manage to do this on your indexer?
If so, could you post an example of how you did this?

Thanks....

0 Karma

fernandoandre
Communicator

tobbedahl, were you able to solve your problem? if yes, please also post your solution.

0 Karma

fernandoandre
Communicator

Hi. At the moment we have many filters but I'll leave you with a simple solution.
==props==
[WinEventLog:Security]
TRANSFORMS-set= setnull

==transforms==
[setnull]
REGEX = (?msi)^EventCode=(?!(552|538|576|528|529)\b)
DEST_KEY = queue
FORMAT = nullQueue

The "setnull" entry redirects you to the stanza with that name on transforms.conf. There you apply the regex to filter what you need.

Hope that helps.

Also read this (splunk lifecycle) http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F
And transforms.conf and props.conf documentation.

0 Karma

tgow
Splunk Employee
Splunk Employee

First of all you will not be able to filter on the Universal Forwarder. If you want to filter events on the Windows server then you will need to install a regular/heavy Forwarder. If you want to continue using a UF instead then you will need to modify the config files on the Indexer. Here is a link to information on how to install a Regular/Heavy Forwarder:

http://docs.splunk.com/Documentation/Splunk/4.2.5/Deploy/Deployaforwarder

0 Karma

tgow
Splunk Employee
Splunk Employee

First of all you will not be able to filter on the Universal Forwarder. If you want to filter events on the Windows server then you will need to install a regular/heavy Forwarder. If you want to continue using a UF instead then you will need to modify the config files on the Indexer. Here is a link to information on how to install a Regular/Heavy Forwarder:

http://docs.splunk.com/Documentation/Splunk/4.2.5/Deploy/Deployaforwarder

marycordova
SplunkTrust
SplunkTrust

Is the universal forwarder good for anything? You cant use for python scripted inputs, you cant filter garbage logs, you cant perform many props/transforms actions, and it isn't even built to run as non-root but still be able to monitor the system it is running on. Basically its just a giant hose built to run up your Splunk license.

@marycordova
0 Karma

fernandoandre
Communicator

unfortunately that's not an option...smallest footprint possible. Thank you for the help.

0 Karma

gekoner
Communicator

Or as tgow said, you can install a regular/heavy Forwarder on the server sending the alerts.

0 Karma

fernandoandre
Communicator

I didn't remember of the following information/table:
http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Typesofforwarders#Forwarder_comparison
In fact, it states that the universal forwarder neither does Event parsing or Per-event filtering.
I will try to do the filtering on the indexer but previous to the indexing process.

0 Karma

gekoner
Communicator

fernandoandre, If I can make a friendly suggestion. Splunk is architect-ed to capture/index ALL data and have you use search queries and other methods to get/view the data you want.
To do this remove your REGEX and FORMAT on your Universal Forwarder client (or remove transform.conf altogether). Restart your UFC.
Then search for the data you are looking for on your search query Web interface.
Ex. = index=* source=WinEventLog:Security AND EventCode=552

If you are looking to be alerted when this occurs you can set up a saved search and a alert to let you know.

Now if you are looking to filter data based on your available bandwidth or license constraints, that's a different story, you can do that. It is just a lot more work.

0 Karma

rshoward
Path Finder

"Splunk is architect-ed to capture/index ALL data"... how convenient for a company that charges per GB indexed per day. Universal Forwarders should allow a regex filter in monitor stream. It only makes sense and doesn't require "parsing" or any Splunk specific technology.

fernandoandre
Communicator

I know it goes against Splunk logic but on this particular case this is really what I need since I only want some events (for example successful/failed logons). The reason for this are simple: 1) I have a very big infrastructure; 2)I want to reduce the traffic on the network. 3) I don't want to index data that I don't need and I don't want to analyze.
My goal is to filter the events at u.forwarder. And believe me, I will receive tons of events on this way.

Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...