Using a Heavy/Light Weight forwarder in this context would insulate you against problems with an individual indexer, but it would not be a good architecture for properly load balancing across your indexers. The forwarder will only rebalance to a new indexer every 30 seconds by default, meaning if you have only one forwarder, you'll only be intermittently loading data into each indexer. You could create a pool of 1:1 relationship of forwarders to indexers and that would ensure all indexers were being fed data at any given time. At that point, you're not gaining a lot from the forwarder but a smallish buffer (by default) in case the indexer is down.
A better architecture would be to put a proper load balancer (F5, NetScaler, Cascade, nginx/apache in reverse proxy mode) that will offer intelligence load balancing and put that in front of splunkd. The load balancer will have a number of available options, such as least connections, best response times, or simple round robin type load balancing methods of distributing load. It can also be setup with a healthcheck to ensure Splunk is up and responding before sending any new connections to splunkd.
... View more
