Deployment Architecture

Can you help me with my routing and filtering question?

central1
Explorer

I have a requirement to send certain filtered log events on to a 3rd party in addition to indexing the events locally.

I've followed this example http://docs.splunk.com/Documentation/Splunk/7.2.0/Forwarding/Routeandfilterdatad#Replicate_a_subset_...

My solution works great. However, I don't quite understand why. To use Splunk's example, why don't I see duplicate (SYSTEM|CONFIG|THREAT) events in my local indexer?? Doesn't the 'routeAll' transform get hit for every event and therefore forwards all events to the 'Everything' destinations? Why does the 'routeSubset' transform need to forward matching events to both 'Subsidiary' AND 'Everything'?

props.conf
[syslog]
TRANSFORMS-routing = routeAll, routeSubset

transforms.conf
[routeAll]
REGEX=(.)
DEST_KEY=_TCP_ROUTING
FORMAT=Everything

[routeSubset]
REGEX=(SYSTEM|CONFIG|THREAT)
DEST_KEY=_TCP_ROUTING
FORMAT=Subsidiary,Everything
Tags (1)
0 Karma
1 Solution

DalJeanis
Legend

The two transforms happen in order.

The first one sets the value of _TCP_ROUTING to Everything.

The second one overrides the value of _TCP_ROUTING for only those items that match the REGEX, and sets it to both Subsidiary and Everything (two places). There is no duplication, because the prior instruction is overwritten, not added to.

This is similar to the various examples that either route everything to NULLQUEUE and then override what you want to keep, or route everything normally and then override only the ones you want to blacklist. The difference is that the override is sending the event both places.

View solution in original post

DalJeanis
Legend

The two transforms happen in order.

The first one sets the value of _TCP_ROUTING to Everything.

The second one overrides the value of _TCP_ROUTING for only those items that match the REGEX, and sets it to both Subsidiary and Everything (two places). There is no duplication, because the prior instruction is overwritten, not added to.

This is similar to the various examples that either route everything to NULLQUEUE and then override what you want to keep, or route everything normally and then override only the ones you want to blacklist. The difference is that the override is sending the event both places.

central1
Explorer

My confusion then comes in how I understand the two transforms are used. I thought each transform is applied to each event. You're saying they are NOT. If an event matches the 'routeSubset' transform, it won't be evaluated against the 'routeAll' transform?? Isn't ordering important? How does the ordering of the transforms in the props.conf file work then?

0 Karma

central1
Explorer

I get it now. We're building a single transform stanza out of the two based on ordering and precedence rules. Thank you!!!! Your help made all the difference.

DalJeanis
Legend

happy to help.

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...