Getting Data In

Setting heavy forwarder for sending only few events which matched with Regx

moohkhol
New Member

I am setting up heavy forwarder on multiple machine, out of that one of them have below requirement,

1) Heavy forwarder should forward sub-set of events which match with given pattern and not all the events.

I search over web, I found couple of Q&A regarding the same but it’s seem not working for me.
First solutions I tried:

http://answers.splunk.com/answers/9076/how-to-configure-a-forwarder-to-filter-and-send-only-the-even...

On the above solution, I can see in forwarder log, it’s continuously saying could not connect with 0.0.0.0:0 and connection failing but nothing is getting forwarded at Indexer side.
Second solution:

I tried with forwarding everything into NullQueue and forward only event which match with REGX by setting up two stanza into transforms.conf.

Seems that above solution also does not work for me.

Pattern which I want to match is following,

oauth.googleusercontent.com, ssl.gstatic.com, fb.com, twitter.com

Heavy forwarder should only send events to indexer if events matched with any of the above patterns.

Please let me know, if you have any question regarding the same.
I have already spend one full day in exploring above but could not find solution.

Sorry to raise this question in that manner but i feel very sad after investigating over a day and could not find solution, Below are the configuration i am suing at my heavy forwarder side.

props.conf

[host::]
TRANSFORMS-set =setnull, allowtheseevents

transform.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[allowtheseevents]
REGEX = (?m)ssl.gstatic.com
FORMAT = splunkindexer_9997

inputs.conf
[monitor:///usr/local/logfilename.log]
index = main
sourcetype = filtersplunkproxy
blacklist = .gz

output.conf
[tcpout]
defaultGroup = splunkindexer_9997

[tcpout:splunkindexer_9997]
server = :9997

[tcpout-server://:9997]
Please correct me if I am doing something wrong here.

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

The essential problem lies in the [allowtheseevents] transform stanza. The FORMAT shall be indexQueue, not a reference to some outputs group.

http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Routeandfilterdatad#Keep_specific_event...

Then you'll need to make your regex include more patterns.

/K

kristian_kolb
Ultra Champion

REGEX = (oauth\.googleusercontent|ssl\.gstatic|fb|twitter)\.com

Should work.

/k

0 Karma

moohkhol
New Member

Thanks Kristian, seems to be working this for me. If you don't mind, Please help me to have generate REGX for below pattern for OR condition with any one of oauth.googleusercontent.com,
ssl.gstatic.com,
fb.com,
twitter.com in anywhere in the events.

0 Karma

kristian_kolb
Ultra Champion

oh, and you need a DEST_KEY as well there;

[allowtheseevents]
REGEX = your regex
FORMAT = indexQueue
DEST_KEY = queue

/k

0 Karma

moohkhol
New Member

I have updated questions with config.

0 Karma

Ayn
Legend

Asking us to do all the work for you is in my eyes pretty rude. Provide us with info on your nullQueue setup with relevant sections from the conf files you were talking about yourself, then we can have a look and see where you might be going wrong.

Get Updates on the Splunk Community!

Splunk Mobile: Your Brand-New Home Screen

Meet Your New Mobile Hub  Hello Splunk Community!  Staying connected to your data—no matter where you are—is ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Enterprise Security (ES) Essentials 8.3 is Now GA — Smarter Detections, Faster ...

As of today, Enterprise Security (ES) Essentials 8.3 is now generally available, helping SOC teams simplify ...