Your forwarding architecture, while complex and difficult to troubleshoot, is technically workable. Sometimes, you have to do what you have to do in a highly segmented network with overly(?) restrictive firewall policies.
Unless you have advanced routing/filtering requirements, you don't need a heavy forwarder anywhere; universal forwarders can do it, and do it more efficiently.
You need to take care not to introduce any bottleneck that will affect your event distribution across the indexer tier. The last forwarding tier needs to have at least 2x forwarders (or ingestionPipelines) than you have indexers to ensure indexers are served concurrently and as evenly as possible.
Your first intermediary tier needs to be properly sized as well to handle the 9000 endpoints (2 TCP connections each, one heartbeat, one data). Make sure you properly set your ulimits such that your intermediary forwarders can handle the large connection counts.
If your intermediary forwarder servers are equipped with enough cores, you can safely increase parallelIngestionPipelines to have each forwarder process multiple inbound/outbound connections in parallel. See the documentation for details. If you have eight-core servers, you can safely configure four parallel pipelines.
Also, ensure you have sufficient network bandwidth on ingress and egress to handle the data volume going through your intermediary forwarding tier. And don't introduce single points of failure into your intermediary forwarding architecture, i.e. have at least 2 servers at each tier.
Hope that helps.
... View more
