Can you help me with my routing and filtering question?

[central1]

I have a requirement to send certain filtered log events on to a 3rd party in addition to indexing the events locally.

I've followed this example http://docs.splunk.com/Documentation/Splunk/7.2.0/Forwarding/Routeandfilterdatad#Replicate_a_subset_...

My solution works great. However, I don't quite understand why. To use Splunk's example, why don't I see duplicate (SYSTEM|CONFIG|THREAT) events in my local indexer?? Doesn't the 'routeAll' transform get hit for every event and therefore forwards all events to the 'Everything' destinations? Why does the 'routeSubset' transform need to forward matching events to both 'Subsidiary' AND 'Everything'?

props.conf
[syslog]
TRANSFORMS-routing = routeAll, routeSubset

transforms.conf
[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything

[DalJeanis · Accepted solution]

The two transforms happen in order.

The first one sets the value of _TCP_ROUTING to Everything.

The second one overrides the value of _TCP_ROUTING for only those items that match the REGEX, and sets it to both Subsidiary and Everything (two places). There is no duplication, because the prior instruction is overwritten, not added to.

This is similar to the various examples that either route everything to NULLQUEUE and then override what you want to keep, or route everything normally and then override only the ones you want to blacklist. The difference is that the override is sending the event both places.

[central1]

My confusion then comes in how I understand the two transforms are used. I thought each transform is applied to each event. You're saying they are NOT. If an event matches the 'routeSubset' transform, it won't be evaluated against the 'routeAll' transform?? Isn't ordering important? How does the ordering of the transforms in the props.conf file work then?

[central1]

I get it now. We're building a single transform stanza out of the two based on ordering and precedence rules. Thank you!!!! Your help made all the difference.

[DalJeanis]

happy to help.