DATA filtering using Heavy forwarders

Venkat_16 — Tue, 29 Sep 2020 17:01:36 GMT

i was tyring to filter a set of data to indexer by filtering out few data and below are the sample logs and configurations:

Here trying to pass only category_id=FLOWERS to the indexer and ignore GIFTS events.
sample log:

177.23.21.50 - - [24/Jul/2014:03:42:00] "GET /flower_store/category.screen?category_id=GIFTS HTTP/1.1" 200 10591 "http://mystore.splunk.com/flower_store/main\\.screen&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 2035 1226

233.77.49.46 - - [24/Jul/2014:03:41:46] "GET /flower_store/product.screen?product_id=K9-BD-01 HTTP/1.1" 200 10560 "http://mystore.splunk.com/flower_store/category.screen?category_id=GIFTS&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 661 1822

177.23.21.50 - - [24/Jul/2014:03:42:00] "GET /flower_store/category.screen?category_id=FLOWERSHTTP/1.1" 200 10591 "http://mystore.splunk.com/flower_store/main\\.screen&JSESSIONID=SD2SL2FF7ADFF5" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 2035 1226

Configuration:
inputs.conf

[monitor:///opt/log/willwork.log]
sourcetype = access_common
index=heavy

outputs.conf

[tcpout]
defaultGroup = my_search_peers
forwardedindex.filter.disable = true
indexAndForward = false

[tcpout:my_search_peers]
server=indexerip:9997

[monitor:///opt/log/willwork.log]
sourcetype = access_common
index=heavy

props.conf
[access-combine]
TRANSFORMS-routing=accessrouting

transforms.conf
[accessrouting]
REGEX=FLOWERS
DEST_KEY=_TCP_ROUTING
FORMAT=my_search_peers

data is getting indexer but GIFTS even is also getting indexed

Re: DATA filtering using Heavy forwarders

jcrabb_splunk — Tue, 28 Nov 2017 14:20:29 GMT

You will want to look at the documentation here:

http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Forwarding/Routeandfilterdatad#Keep_specific_events_and_discard_the_rest

Essentially you will want to route the unwanted data to the nullqueue.

topic Re: DATA filtering using Heavy forwarders in Getting Data In

DATA filtering using Heavy forwarders

Re: DATA filtering using Heavy forwarders