Let's distinguish "intermediary forwarder" from heavy forwarder. You can easily use a UF to serve as an intermediary forwarder, if you don't have requirements that mandate a heavy forwarder. And that's what you should use, if no such requirements exist, because the UF has a much lower resource impact, scales better and has less overhead on the wire.
As a rule of thumb, filtering events before they go over the wire is only going to provide a net benefit if a large number of events are subject to be filtered (~40-50%). Otherwise, do your filtering on the indexers and take advantage of the more efficient wire protocol the UF has.
Personally I think that there are more disadvantages and pitfalls with any architecture that contains intermediary forwarding tiers than there are advantages. The biggest issue that often occurs is that intermediary forwarding tiers are not properly sized to not impact event distribution of 100s or 1000s of endpoint forwarders across indexers. You want to have at least a 2x intermediary forwarding pipelines to indexer ratio to ensure that most indexers receive data at any given time.
Troubleshooting also gets more complicated and you have another tier that you need to manage configurations on.
Sometimes you have to have an intermediary forwarding tier, for example if network restrictions don't permit connections from forwarders directly to indexers, or if you need to do selective forwarding to third-party systems.
If you can avoid them, you are almost always better off and end up with an environment that has less event distribution issues, lower data ingest latency and is easier to manage and provides you with a better TCO overall, since you need fewer servers to support your architecture.
I hope this helps!
... View more