topic heavy forwarder to route base on _raw values in Getting Data In

heavy forwarder to route base on _raw values

gooza — Thu, 23 Feb 2012 11:53:26 GMT

I'm trying to use the heavy forwarder to route data to different indexes based on values in _raw , is this possible ?

the configuration files are:

inputs.conf:

[tcp://9997]

sourcetype = FromFooandBarbysinglehost

props.conf:

[FromFooandBarbysinglehost]

BREAK_ONLY_BEFORE = ^

TRANSFORMS-routing = FromFoo,FromBar

transforms.conf:

[FromFoo]

REGEX = (?i) From|.+?Foo

DEST_KEY = _TCP_ROUTING

FORMAT = outtoFoo

[FromBar]

REGEX = (?i) From|.+?Bar

DEST_KEY = _TCP_ROUTING

FORMAT = outtoBar

outputs.conf:

[tcpout:outtoFoo]

server = 10.10.10.10:1111

sendCookedData = false

[tcpout:outtoBar]

server = 10.10.10.10:2222

sendCookedData = false

in the indexer 10.10.10.10 the TCP port 1111 is indexed to foo index , and 2222 is indexed to Bar index)

me problem is that I see both foo data and bar data in both indexes , it is like there is no termination to the transforms process and both are sent to both ports.

I double checked my REGEX in the search bar in splunk and it does show only the relevant data

what am I missing ?

Re: heavy forwarder to route base on _raw values

gkanapathy — Thu, 23 Feb 2012 16:55:46 GMT

when your data arrives in the indexers, what is the sourcetype? Is it FromFooandBarbysinglehost?

Re: heavy forwarder to route base on _raw values

gooza — Fri, 24 Feb 2012 07:24:09 GMT

no , the tcpout redirect the data to different TCP ports on the indexer and each port has its own source type and index

Re: heavy forwarder to route base on _raw values

Jason — Mon, 16 Apr 2012 16:04:26 GMT

_TCP_ROUTING is used to send data to another indexer. If the data is already at the indexer you want, just use the _MetaData:Index key to change the index for that event.

[FromFoo]
REGEX = (?i) From|.+?Foo
DEST_KEY = _MetaData:Index
FORMAT = foo_index

[FromBar]
REGEX = (?i) From|.+?Bar
DEST_KEY = _MetaData:Index
FORMAT = bar_index