topic Re: Setting heavy forwarder for sending only few events which matched with Regx in Getting Data In

Setting heavy forwarder for sending only few events which matched with Regx

moohkhol — Wed, 26 Feb 2014 07:02:52 GMT

I am setting up heavy forwarder on multiple machine, out of that one of them have below requirement,

1) Heavy forwarder should forward sub-set of events which match with given pattern and not all the events.

I search over web, I found couple of Q&A regarding the same but it’s seem not working for me.
First solutions I tried:

http://answers.splunk.com/answers/9076/how-to-configure-a-forwarder-to-filter-and-send-only-the-events-i-want

On the above solution, I can see in forwarder log, it’s continuously saying could not connect with 0.0.0.0:0 and connection failing but nothing is getting forwarded at Indexer side.
Second solution:

I tried with forwarding everything into NullQueue and forward only event which match with REGX by setting up two stanza into transforms.conf.

Seems that above solution also does not work for me.

Pattern which I want to match is following,

oauth.googleusercontent.com, ssl.gstatic.com, fb.com, twitter.com

Heavy forwarder should only send events to indexer if events matched with any of the above patterns.

Please let me know, if you have any question regarding the same.
I have already spend one full day in exploring above but could not find solution.

Sorry to raise this question in that manner but i feel very sad after investigating over a day and could not find solution, Below are the configuration i am suing at my heavy forwarder side.

props.conf

[host::]
TRANSFORMS-set =setnull, allowtheseevents

transform.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[allowtheseevents]
REGEX = (?m)ssl.gstatic.com
FORMAT = splunkindexer_9997

inputs.conf
[monitor:///usr/local/logfilename.log]
index = main
sourcetype = filtersplunkproxy
blacklist = .gz

output.conf
[tcpout]
defaultGroup = splunkindexer_9997

[tcpout:splunkindexer_9997]
server = :9997

[tcpout-server://:9997]
Please correct me if I am doing something wrong here.

Re: Setting heavy forwarder for sending only few events which matched with Regx

Ayn — Wed, 26 Feb 2014 07:56:58 GMT

Asking us to do all the work for you is in my eyes pretty rude. Provide us with info on your nullQueue setup with relevant sections from the conf files you were talking about yourself, then we can have a look and see where you might be going wrong.

Re: Setting heavy forwarder for sending only few events which matched with Regx

moohkhol — Wed, 26 Feb 2014 09:12:46 GMT

I have updated questions with config.

Re: Setting heavy forwarder for sending only few events which matched with Regx

kristian_kolb — Wed, 26 Feb 2014 09:58:42 GMT

The essential problem lies in the [allowtheseevents] transform stanza. The FORMAT shall be indexQueue, not a reference to some outputs group.

http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Then you'll need to make your regex include more patterns.

Re: Setting heavy forwarder for sending only few events which matched with Regx

kristian_kolb — Wed, 26 Feb 2014 10:00:33 GMT

oh, and you need a DEST_KEY as well there;

[allowtheseevents]
REGEX = your regex
FORMAT = indexQueue
DEST_KEY = queue

Re: Setting heavy forwarder for sending only few events which matched with Regx

moohkhol — Wed, 26 Feb 2014 10:51:34 GMT

Thanks Kristian, seems to be working this for me. If you don't mind, Please help me to have generate REGX for below pattern for OR condition with any one of oauth.googleusercontent.com,
ssl.gstatic.com,
fb.com,
twitter.com in anywhere in the events.

Re: Setting heavy forwarder for sending only few events which matched with Regx

kristian_kolb — Wed, 26 Feb 2014 12:07:43 GMT

REGEX = (oauth\.googleusercontent|ssl\.gstatic|fb|twitter)\.com

Should work.