Currently, we've got four indexers and about 1400 forwarders. These forwarders all send their data to the indexers over the Internet.
The autoLB parameter is set on the forwarders to send to all four indexers. I'm curious if there is any way to set up a reverse proxy so that we can have one externally facing IP address and port, but then have the incoming traffic divided up between the indexers.
Many of our forwarders are at customer sites, and having them update their firewall rules every time we add a new indexer is frustrating for the customer and cumbersome for us.
Similar to what Ayn is suggesting I might do something like this:
This gives your customers a simple internal network / firewall configuration (at the cost of a couple of VMs), and gives you the ability to change your indexer footprint more or less at will.
I should mention that this is precisely how we've set it up, and it seems we're running a similar service (architecture wise) to what is described in the initial question, and it works perfectly.
If the intermediate forwarder is an Universal Forwarder, it has some default limits set, such as that it will send data at a maximum speed of 256kBps. That's just a default value though, that can easily be changed, as are all others. Heavy forwarders have no limits like this set by default. Either way, it's no bottleneck because it's essentially just another Splunk instance - just one that happens to forward stuff instead of indexing stuff.