Getting Data In

How can I filter events before they are indexed so they aren't indexed?

amit2301
New Member

I tried this solution but no success.
I am trying to filter data from being indexed.I need only the Error events

In props conf:
[source:://C:\Windows\System32\winevt\Logs]

Transforms must be applied in this order

to make sure events are dropped on the

floor prior to making their way to the

index processor

TRANSFORMS-set = setnull, setparsing

In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = Error
DEST_KEY = queue
FORMAT = indexQueue

0 Karma

woodcock
Esteemed Legend

If you are sure that your settings are correct, it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

0 Karma

vhharanpositka
Path Finder

Hi amit

I have tried the above configuration but still, I have a thought that the events in pushed to the null queue are added in the license usage. Please, someone, make sure that it won't be added in the License usage

0 Karma

gaurav_maniar
Builder

Hi Amit,

props.conf & transforms.conf configuration looks correct.

Parsing of data can only be done before indexing i.e at Forwarder side.
So you need to put these conf files on Forwarder instance, and it should be a Heavy Forwarder.

Please upvote and accept the answer if it solves your issue.

Happy Splunking !!!

0 Karma

rashi83
Path Finder

As per the above link, filtering can be done on Indexer level as well.

Isn't correct ?

0 Karma

gaurav_maniar
Builder

Yes, but most of the time using Universal Forwarder and parsing at Indexer causes issues.

0 Karma

rashi83
Path Finder

Can you elaborate what are such issues?

We have a situation where we do not control forwarders and there is huge amount of unwanted logs coming and outgrowing the licenses. We have a demand to control such logs are Indexer level.

0 Karma

rashi83
Path Finder

Can you please confirm location of props.conf and transform.conf ? I am trying to filter the events on Indexer as I do not have access to forwarders.

0 Karma

sbbadri
Motivator

@amit2301
Your props.conf and transforms.conf seems correct.

Please double check you have proper regex for indexQueue. Also check you must place your props.conf and transforms.conf on the indexer or heavy forwarder.

Also go through below link for further details,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...