Getting Data In
Highlighted

DATA filtering using Heavy forwarders

Contributor

i was tyring to filter a set of data to indexer by filtering out few data and below are the sample logs and configurations:

Here trying to pass only category_id=FLOWERS to the indexer and ignore GIFTS events.
sample log:

177.23.21.50 - - [24/Jul/2014:03:42:00] "GET /flowerstore/category.screen?categoryid=GIFTS HTTP/1.1" 200 10591 "http://mystore.splunk.com/flower_store/main\\.screen&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 2035 1226

233.77.49.46 - - [24/Jul/2014:03:41:46] "GET /flowerstore/product.screen?productid=K9-BD-01 HTTP/1.1" 200 10560 "http://mystore.splunk.com/flower_store/category.screen?category_id=GIFTS&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 661 1822

177.23.21.50 - - [24/Jul/2014:03:42:00] "GET /flowerstore/category.screen?categoryid=FLOWERSHTTP/1.1" 200 10591 "http://mystore.splunk.com/flower_store/main\\.screen&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 2035 1226

Configuration:
inputs.conf

[monitor:///opt/log/willwork.log]
sourcetype = access_common
index=heavy

outputs.conf

[tcpout]
defaultGroup = mysearchpeers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:mysearchpeers]
server=indexerip:9997

[monitor:///opt/log/willwork.log]
sourcetype = access_common
index=heavy

props.conf
[access-combine]
TRANSFORMS-routing=accessrouting

transforms.conf
[accessrouting]
REGEX=FLOWERS
DESTKEY=TCPROUTING
FORMAT=my
search_peers

data is getting indexer but GIFTS even is also getting indexed

0 Karma
Highlighted

Re: DATA filtering using Heavy forwarders

Splunk Employee
Splunk Employee

You will want to look at the documentation here:

http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_...

Essentially you will want to route the unwanted data to the nullqueue.

Jacob
Sr. Technical Support Engineer
0 Karma