Discard events on forwarder/Indexer which includes specific tags in Logs

[nikhilagrawal]

Hi
I want to discard log lines which includes specific tag "reversed position attributes
" in log file.

We want to avoid lines which includes these specific tags and index rest of the log file. Is it possible to discard these log lines on forwarder level?

Please suggest suitable solution. If you need more information please let me know.

Thanks.

[strive]

Define the regex in transforms.conf and use the transformation in props.conf

Say, for example if you want to strip header. Your header starts with #Fields.......

Add following stanza in transforms.conf

[strip_header]  
REGEX = #Fields  
DEST_KEY = queue  
FORMAT = nullQueue  

In props.conf

[your_stanza]  
TRANSFORMS-tonull = strip_header 

[jeremiahc4]

Did you see MuS's comment previously? Are you doing this on an indexer or heavy forwarder (i.e. a full splunk install)?

"Regarding the forwarder, this can only be done on a forwarder which does data parsing. Universal forwarder cannot/will not parse data. To do so, you would need to setup a so called heavy forwarder. "

[nikhilagrawal]

I have created the filed and used in transform.conf but it does not seems to be working. Field Name=Test
Test="Ndi reversed position attributes:"
[strip_header]
REGEX = Test
DEST_KEY = queue
FORMAT = nullQueue

props.conf:
[source::/opt/splunk_fx_prod/fx_hot_bucket/fxmp_hedger_prod/.../*]
TRANSFORMS-null= strip_header

Can someone help here please?

[nikhilagrawal]

Thanks for response.We have log pattern as below and I want to discard all events which includes "Ndi reversed position attributes" string and log the rest of the log file.

2014-04-14 12:29:04,394 INFO [com.rbsfm.fxmicropay.netpositioner.ndi.NdiPositionPublisher] Ndi reversed position attributes: CNH:-753.21:GBP:70.25:FXMPMALN:16-Apr-2014 00:00:00:0

So Can I use something like this?

[strip_header]

REGEX = "Ndi reversed position attributes "
DEST_KEY = queue

FORMAT = nullQueue

[MuS]

Hi nikhilagrawal,

First of all, this is possible in Splunk just follow the docs about route and filter data.

Regarding the forwarder, this can only be done on a forwarder which does data parsing. Universal forwarder cannot/will not parse data. To do so, you would need to setup a so called heavy forwarder. This setup is able to do data parsing and could route and / or filter data before sending to the indexer.
Nevertheless the indexer itself is also able to parse data, so you can also setup route and filter on the indexer.

hope this helps ...

cheers, MuS