Forwarders and Reverse Proxy

[mzorzi]

Currently, we've got four indexers and about 1400 forwarders. These forwarders all send their data to the indexers over the Internet.

The autoLB parameter is set on the forwarders to send to all four indexers. I'm curious if there is any way to set up a reverse proxy so that we can have one externally facing IP address and port, but then have the incoming traffic divided up between the indexers.

Many of our forwarders are at customer sites, and having them update their firewall rules every time we add a new indexer is frustrating for the customer and cumbersome for us.

[dwaddle]

Similar to what Ayn is suggesting I might do something like this:

1. Set up a pair "bridgehead" light forwarders at each customer location (or require the customer to do so). Their local systems all forward to these.
2. Also set up two (maybe more, always in pairs) heavy forwarders at your own location that are accessible by the customer bridgehead forwarders. The customer bridgeheads forward to your edge forwarders who can then offload parsing from your indexers and pass pre-parsed events on to your indexer farm.

This gives your customers a simple internal network / firewall configuration (at the cost of a couple of VMs), and gives you the ability to change your indexer footprint more or less at will.

[Ayn]

I should mention that this is precisely how we've set it up, and it seems we're running a similar service (architecture wise) to what is described in the initial question, and it works perfectly.

[Ayn]

If the intermediate forwarder is an Universal Forwarder, it has some default limits set, such as that it will send data at a maximum speed of 256kBps. That's just a default value though, that can easily be changed, as are all others. Heavy forwarders have no limits like this set by default. Either way, it's no bottleneck because it's essentially just another Splunk instance - just one that happens to forward stuff instead of indexing stuff.

[andru]

If intermediate forwarders are used, do you know of any limitations? For example, how many forwarders could connect to a single intermediate forwarder before bottle necking would occur?

[Ayn]

Did you consider intermediate forwarders that use autoLB against backend indexers? If yes, what made you not choose that option?