Our network has 4 "zones". In general, servers in each zone can only talk to other servers in the same zone as them. As such, we have a Splunk indexer in each zone, which should be receiving input from all the forwarders in it's zone. All forwarders are how ever able to talk to a single Deployment Server. At present, I am pushing (via the Deployment Server) a different outputs.conf file to the forwarders in each zone, directing them to send their data to the zone-specific indexer. I'd like to simplify this by pushing only one outputs.conf, which would include all 4 indexers in it, and allow the forwarder to make the decision on which to use based on which one it can reach.
I believe I can do this easily by configuring all 4 indexers in an outputs.conf file and allowing the forwarder to replicate data among all 4. Obviously, only one indexer will ever actually receive the data. I'm concerned that this however will produce a bunch of unneeded network traffic, firewall log events, and Splunk errors as the forwarder keeps retrying indexers it cannot reach. Is there a better way to achieve this goal?
The solution you have already may be the most simple.
One I might suggest would be basically IP anycast. On each indexer, put the same aliased IP on the loopback - say 10.255.255.1. Then, configure every host in every zone to send data to 10.255.255.1. It is then a question of network routing in each zone to send packets for 10.255.255.1 to the "local" 10.255.255.1 for that zone.
If each zone has its own isolated dynamic routing -- that is routers in zone A cannot see OSPF/ISIS/EIGRP/RIP routes from zone B/C/D -- this is fairly easy to set up. If all of your zones have a common dynamic routing, it will be much more difficult and you'll need to discuss with your local LAN people.
I'm thinking it may be possible with some clever adjustments to the Backoff Settings in outputs.conf to get this to be mroe efficient.