How can I filter events before they are indexed so they aren't indexed?

[amit2301]

I tried this solution but no success.
I am trying to filter data from being indexed.I need only the Error events

In props conf:
[source:://C:\Windows\System32\winevt\Logs]

Transforms must be applied in this order

to make sure events are dropped on the

floor prior to making their way to the

index processor

TRANSFORMS-set = setnull, setparsing

In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = Error
DEST_KEY = queue
FORMAT = indexQueue

[woodcock]

If you are sure that your settings are correct, it must be something else. If you are doing a sourcetype override/overwrite, you must use the ORIGINAL value, NOT the new value. You must deploy your settings to the first full instance(s) of Splunk that handle the events (usually either the HF tier if you use one, or else your Indexer tier) UNLESS you are using HEC's JSON endpoint (it gets pre-cooked) or INDEXED_EXTRACTIONS (configs go on the UF in that case), then restart all Splunk instances there. When (re)evaluating, you must send in new events (old events will stay broken), then test using _index_earliest=-5m to be absolutely certain that you are only examining the newly indexed events.

[vhharanpositka]

Hi amit

I have tried the above configuration but still, I have a thought that the events in pushed to the null queue are added in the license usage. Please, someone, make sure that it won't be added in the License usage

[gaurav_maniar]

Hi Amit,

props.conf & transforms.conf configuration looks correct.

Parsing of data can only be done before indexing i.e at Forwarder side.
So you need to put these conf files on Forwarder instance, and it should be a Heavy Forwarder.

Please upvote and accept the answer if it solves your issue.

Happy Splunking !!!

[rashi83]

As per the above link, filtering can be done on Indexer level as well.

Isn't correct ?

[gaurav_maniar]

Yes, but most of the time using Universal Forwarder and parsing at Indexer causes issues.

[rashi83]

Can you elaborate what are such issues?

We have a situation where we do not control forwarders and there is huge amount of unwanted logs coming and outgrowing the licenses. We have a demand to control such logs are Indexer level.

[rashi83]

Can you please confirm location of props.conf and transform.conf ? I am trying to filter the events on Indexer as I do not have access to forwarders.

[sbbadri]

@amit2301
Your props.conf and transforms.conf seems correct.

Please double check you have proper regex for indexQueue. Also check you must place your props.conf and transforms.conf on the indexer or heavy forwarder.

Also go through below link for further details,
http://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad