From my understanding, the param `defaultGroup` under the stanza `[tcpout]` in `outputs.conf` can be set to a comma-separated list based on what's defined in `[tcpout:<groupn>]` stanzas, i.e.:
defaultGroup = group1, group2, group3, group4
Okay. But when we define `outputs.conf` like this, the forwarder will route all traffic to the target servers in all groups defined in that param `defaultGroup`.
How do we define a "backup" group where in the traffic is rerouted to the "next" group in case the default groups aren't reachable?
Those aren't indexers. They are intermediate heavyforwarders that represent regions. I.e.
# North America UF Outputsconf
defaultGroup = NA
[tcpout:EMEA] server = mycompany-emea-hfa:9997, mycompany-emea-hfb:9997
[tcpout:APAC] server = mycompany-apac-hfa:9997, mycompany-apac-hfb:9997
[tcpout:NA] server = mycompany-na-hfa:9997, mycompany-na-hfb:9997
[tcpout:LATAM] server = mycompany-latam-hfa:9997, mycompany-latam-hfb:9997
These intermediate HFs all output to SplunkCloud. I was thinking how I could route traffic to LATAM if say, the servers in NA are down.
Actually there is one way to do this, but it requires that you are using multisite cluster with indexes discovery feature enabled. Then you can just add on your CM:
splunk edit cluster-config -forwarder_site_failover site1:site2
Rather that use multiple groups of one indexer each, use a single group of multiple indexers. The UF will send to each indexer in turn, skipping those that are not available.
[tcpout] defaultGroup = group1 [tcpout:group1] server=10.1.1.197:9997,myhost.Splunk.com:9997,foo.Splunk.com:9997
If your indexers are clustered then consider using Indexer Discovery, which achieves the same effect.