I have a requirement to send certain filtered log events on to a 3rd party in addition to indexing the events locally.
I've followed this example http://docs.splunk.com/Documentation/Splunk/7.2.0/Forwarding/Routeandfilterdatad#Replicate_a_subset_...
My solution works great. However, I don't quite understand why. To use Splunk's example, why don't I see duplicate (SYSTEM|CONFIG|THREAT) events in my local indexer?? Doesn't the 'routeAll' transform get hit for every event and therefore forwards all events to the 'Everything' destinations? Why does the 'routeSubset' transform need to forward matching events to both 'Subsidiary' AND 'Everything'?
props.conf
[syslog]
TRANSFORMS-routing = routeAll, routeSubset
transforms.conf
[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything
[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything
The two transforms happen in order.
The first one sets the value of _TCP_ROUTING
to Everything
.
The second one overrides the value of _TCP_ROUTING
for only those items that match the REGEX, and sets it to both Subsidiary
and Everything
(two places). There is no duplication, because the prior instruction is overwritten, not added to.
This is similar to the various examples that either route everything to NULLQUEUE and then override what you want to keep, or route everything normally and then override only the ones you want to blacklist. The difference is that the override is sending the event both places.
The two transforms happen in order.
The first one sets the value of _TCP_ROUTING
to Everything
.
The second one overrides the value of _TCP_ROUTING
for only those items that match the REGEX, and sets it to both Subsidiary
and Everything
(two places). There is no duplication, because the prior instruction is overwritten, not added to.
This is similar to the various examples that either route everything to NULLQUEUE and then override what you want to keep, or route everything normally and then override only the ones you want to blacklist. The difference is that the override is sending the event both places.
My confusion then comes in how I understand the two transforms are used. I thought each transform is applied to each event. You're saying they are NOT. If an event matches the 'routeSubset' transform, it won't be evaluated against the 'routeAll' transform?? Isn't ordering important? How does the ordering of the transforms in the props.conf file work then?
I get it now. We're building a single transform stanza out of the two based on ordering and precedence rules. Thank you!!!! Your help made all the difference.
happy to help.