topic Re: Filter events from forwards in Getting Data In

Filter events from forwards

chittari — Wed, 25 Apr 2012 09:51:38 GMT

Hello - I want to send only events with keyword BIDPRICE from my application logs. I guess i need to modifiy props.conf, transforms.conf and outputs.conf

Can someone help me what changes are acually needed.
fyi-Currently forwading is working fine without this filter.

Thanks in advance for help.

Re: Filter events from forwards

Ayn — Wed, 25 Apr 2012 09:58:02 GMT

If these are light or universal forwarder, you cannot filter the logs there. You'll have to do it at the indexer instead. Instructions on how to do this are available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad#Filter_event_data_and_send_to_queues

Re: Filter events from forwards

kristian_kolb — Mon, 28 Sep 2020 11:43:35 GMT

This is quite thoroughly discussed in the docs, please see:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

If you have a heavy forwarder, the settings should be placed there. In all other cases, the settings should be on the indexer.

In short, what you need to do is

In props.conf:

[your_applog_sourcetype] TRANSFORMS-keep_only_bidprice= setnull,setparsing
In transforms.conf:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue

[setparsing]
REGEX = BIDPRICE
DEST_KEY = queue
FORMAT = indexQueue

UPDATE:

What this does is to apply a set of rules to your incoming data. First, in props.conf you tell splunk that all events of a certain sourcetype should pass through a transform (or in this case, two transforms - setnull and setparsing).

Then, in transforms.conf, you state how data should be treated. setnull routes data to the nullQueue, i.e. throws it away. setparsing will match all events containing the string BIDPRICE and send these on for parsing and indexing.

If this does not work for you, then please tell us more about your setup, and post the relevant parts of props.conf and transforms.conf, along with a few sample events from your application log file.

You should know though that this will only work for new data coming in, and not alter any existing events already in your index.

UPDATE AGAIN:

DEST_KEY = queue is where you state what parameter should be altered by the transform.

FORMAT = nullQueue is the value that will be set for the parameter. (nullQueue is something like /dev/null)

Think of it as saying queue = nullQueue.

So what happens when use TRANSFORMS-set = setnull, setparsing in props.conf is;

set the destination of all events to
nullQueue (since all events will
match the dot (.) in the regex, i.e.
throw them away.
then, for those matching the regex in
setparsing, i.e.
BidPrice, the destination should be
re-written to be the indexQueue,
which is where events normally for
indexing.

Hope this helps,

Kristian

Re: Filter events from forwards

chittari — Wed, 25 Apr 2012 10:47:27 GMT

Thanks for your reply But This did not work.. I still see all the events in my indexer (Indexer was restarted). Not sure what below values refers to, Any conf update needed to understand what is queue and nullQueue?
"
DEST_KEY = queue
FORMAT = nullQueue
"

Re: Filter events from forwards

kristian_kolb — Wed, 25 Apr 2012 11:05:37 GMT

see update above. /k

Re: Filter events from forwards

chittari — Mon, 28 Sep 2020 11:43:37 GMT

Sorry for conusion around. Here is my current setup wrt Filtering and i see all events flowing down to Indexer

*props.conf *

[host::sgppsr00346.XXXX.XXXX.com]
TRANSFORMS-set= setnull,setparsing

*transforms.conf *

[setnull]
REGEX = .
DEST_KEY = queue [Not sure what should be here for the case of TCP]
FORMAT = nullQueue [Not sure what should be here for the case of TCP]

[setparsing]
REGEX = [BidPrice]
DEST_KEY = _TCP_ROUTING [TCP routing]
FORMAT = GroupName [This is currect group name as per outputs.conf]

Re: Filter events from forwards

kristian_kolb — Wed, 25 Apr 2012 11:45:39 GMT

What does the event look like? You have to make sure that the regex matches the text in the event. If it does not, then all events are thrown away - since they match the setnull transform.

Re: Filter events from forwards

chittari — Wed, 25 Apr 2012 11:57:44 GMT

Event looks like this

INFO 19:55:55,284 - UST prices for FIDO : [423423] : BidPrice[103.25390625] AskPrice[103.28515625] BidYield[0.7235491957] AskYield[0.71622942792] at [1335354955284]

Re: Filter events from forwards

kristian_kolb — Wed, 25 Apr 2012 12:01:39 GMT

Then the regex in transforms.conf should look like;

REGEX = BidPrice

Re: Filter events from forwards

kristian_kolb — Mon, 28 Sep 2020 11:43:39 GMT

And don't use _TCP_ROUTING unless you know what you are doing.

Re: Filter events from forwards

chittari — Mon, 28 Sep 2020 11:43:42 GMT

Does not work at all if I don't add _TCP_ROUTING. I have added this because document says for TCP routing.(between two Linux servers).

What does desk_key = queue mean?

My setup is still not okay, I think reason is [setnull] options around.

Thanks for your help.

Re: Filter events from forwards

kristian_kolb — Wed, 25 Apr 2012 13:01:49 GMT

Sorry, but WHERE (in which file, on which host) are you making these configuration changes, and what is your setup?

A) Heavy Forwarder -> indexer ?
B) Universal Forwarder -> indexer ?

For the rest, see update above.

Re: Filter events from forwards

chittari — Wed, 25 Apr 2012 14:52:01 GMT

Universal Forwarder -> indexer.

Re: Filter events from forwards

chittari — Wed, 25 Apr 2012 16:15:55 GMT

It works now ... (relief). I guess the problem was within props.conf file. Looks like [host:: XXXX] does not work with universal forwaders. I chaged this to [source:: ] and it worked.

Thanks a lot for your guidence

Re: Filter events from forwards

kristian_kolb — Wed, 25 Apr 2012 18:28:17 GMT

you are welcome 🙂