topic Re: Route and filter data from HEC in Getting Data In

How to route and filter data from HEC?

martaBenedetti — Wed, 11 May 2022 15:41:56 GMT

Hi Community,

I have the need to filter data based on a specific field value and route to a different group of indexers.

Data is coming through HEC configured on a Heavy Forwarder like this:

[http://tokenName] index = main indexes = main outputgroup = my_indexers sourcetype = _json token = <string> source = mysource

I'd like to use props.conf and transforms.conf as suggested here like this:

props.conf [source::mysource] TRANSFORMS-routing=otherIndexersRouting transforms.conf [otherIndexersRouting] REGEX=\"domain\"\:\s\"CARD\" DEST_KEY=_TCP_ROUTING FORMAT=other_indexers

In outputs.conf I'd add the stanza [tcpOut:other_indexers]

Is this possible? Is there another way to achieve this goal?

Thank you

Marta

Re: Route and filter data from HEC

PickleRick — Wed, 11 May 2022 14:07:25 GMT

Close.

You need to specify the stanza in props.conf as

[source::mysource]

Then you can call appropriate transforms from there.

Keep in mind though that the hierarchy is source->host->sourcetype so if you have - for example - your host field overwritten based on data from the raw event in a transform called from sourcetype-based stanza, you won't be able to use this host value as selector.

Re: Route and filter data from HEC

somesoni2 — Wed, 11 May 2022 14:11:41 GMT

The configuration provided in the link should be the way to go. Just make sure that you chose appropriate source ([source::YourSource]) OR sourcetype in props.conf.

Another example:

https://community.splunk.com/t5/Getting-Data-In/How-to-route-and-filter-data-on-the-Heavy-Forwarder-to-separate/m-p/216971

Re: Route and filter data from HEC

martaBenedetti — Wed, 11 May 2022 14:25:21 GMT

I was afraid the solution couldn't work with HTTP event collector since I've only used this configuration with classic monitor inputs.

The source stanza was just a typo, I've corrected it 🙂

Thank you

Marta

Re: Route and filter data from HEC

martaBenedetti — Wed, 11 May 2022 14:25:52 GMT

The source stanza was just a typo, I've corrected it 🙂

Thank you

Marta