How to route and filter data from HEC?

[martaBenedetti]

Hi Community,

I have the need to filter data based on a specific field value and route to a different group of indexers.

Data is coming through HEC configured on a Heavy Forwarder like this:

[http://tokenName]
index = main
indexes = main
outputgroup = my_indexers
sourcetype = _json
token = <string>
source = mysource

I'd like to use props.conf and transforms.conf as suggested here like this:

props.conf
[source::mysource]
TRANSFORMS-routing=otherIndexersRouting

transforms.conf
[otherIndexersRouting]
REGEX=\"domain\"\:\s\"CARD\"
DEST_KEY=_TCP_ROUTING
FORMAT=other_indexers

In outputs.conf I'd add the stanza [tcpOut:other_indexers]

Is this possible? Is there another way to achieve this goal?

Thank you

Marta

[somesoni2 · Accepted solution]

The configuration provided in the link should be the way to go. Just make sure that you chose appropriate source ([source::YourSource]) OR sourcetype in props.conf.

Another example:

https://community.splunk.com/t5/Getting-Data-In/How-to-route-and-filter-data-on-the-Heavy-Forwarder-...

[martaBenedetti]

I was afraid the solution couldn't work with HTTP event collector since I've only used this configuration with classic monitor inputs.

The source stanza was just a typo, I've corrected it 🙂

Thank you

Marta

[PickleRick]

Close.

You need to specify the stanza in props.conf as

[source::mysource]

Then you can call appropriate transforms from there.

Keep in mind though that the hierarchy is source->host->sourcetype so if you have - for example - your host field overwritten based on data from the raw event in a transform called from sourcetype-based stanza, you won't be able to use this host value as selector.

[martaBenedetti]

The source stanza was just a typo, I've corrected it 🙂

Thank you

Marta