topic Re: Discard events on forwarder/Indexer which includes specific tags in Logs in Getting Data In

Discard events on forwarder/Indexer which includes specific tags in Logs

nikhilagrawal — Mon, 14 Apr 2014 10:21:59 GMT

Hi
I want to discard log lines which includes specific tag "reversed position attributes
" in log file.

We want to avoid lines which includes these specific tags and index rest of the log file. Is it possible to discard these log lines on forwarder level?

Please suggest suitable solution. If you need more information please let me know.

Thanks.

Re: Discard events on forwarder/Indexer which includes specific tags in Logs

MuS — Mon, 14 Apr 2014 10:48:55 GMT

Hi nikhilagrawal,

First of all, this is possible in Splunk just follow the docs about route and filter data.

Regarding the forwarder, this can only be done on a forwarder which does data parsing. Universal forwarder cannot/will not parse data. To do so, you would need to setup a so called heavy forwarder. This setup is able to do data parsing and could route and / or filter data before sending to the indexer.
Nevertheless the indexer itself is also able to parse data, so you can also setup route and filter on the indexer.

hope this helps ...

cheers, MuS

Re: Discard events on forwarder/Indexer which includes specific tags in Logs

strive — Mon, 14 Apr 2014 11:22:30 GMT

Define the regex in transforms.conf and use the transformation in props.conf

Say, for example if you want to strip header. Your header starts with #Fields.......

Add following stanza in transforms.conf

[strip_header]  
REGEX = #Fields  
DEST_KEY = queue  
FORMAT = nullQueue

In props.conf

[your_stanza]  
TRANSFORMS-tonull = strip_header

Re: Discard events on forwarder/Indexer which includes specific tags in Logs

nikhilagrawal — Mon, 28 Sep 2020 16:22:47 GMT

Thanks for response.We have log pattern as below and I want to discard all events which includes "Ndi reversed position attributes" string and log the rest of the log file.

2014-04-14 12:29:04,394 INFO [com.rbsfm.fxmicropay.netpositioner.ndi.NdiPositionPublisher] Ndi reversed position attributes: CNH:-753.21:GBP:70.25:FXMPMALN:16-Apr-2014 00:00:00:0

So Can I use something like this?

[strip_header]

REGEX = "Ndi reversed position attributes "
DEST_KEY = queue

FORMAT = nullQueue

Re: Discard events on forwarder/Indexer which includes specific tags in Logs

nikhilagrawal — Mon, 28 Sep 2020 16:35:56 GMT

I have created the filed and used in transform.conf but it does not seems to be working. Field Name=Test
Test="Ndi reversed position attributes:"
[strip_header]
REGEX = Test
DEST_KEY = queue
FORMAT = nullQueue

props.conf:
[source::/opt/splunk_fx_prod/fx_hot_bucket/fxmp_hedger_prod/.../*]
TRANSFORMS-null= strip_header

Can someone help here please?

Re: Discard events on forwarder/Indexer which includes specific tags in Logs

jeremiahc4 — Wed, 21 May 2014 20:40:00 GMT

Did you see MuS's comment previously? Are you doing this on an indexer or heavy forwarder (i.e. a full splunk install)?

"Regarding the forwarder, this can only be done on a forwarder which does data parsing. Universal forwarder cannot/will not parse data. To do so, you would need to setup a so called heavy forwarder. "