How to switch universal forwarder traffic between heavy forwarder and Splunk Cloud directly?

[bpdubs]

I have an interesting scenario that I haven't been able to find any guidance on. We use Splunk Cloud, and we have two heavy forwarders in our network which all our universal forwarders send to.

What I'd like to do is configure a universal forwarder to either send directly to Splunk Cloud, if the device is outside our network, or send to a heavy forwarder if it is in our network. This would be for devices like laptops that might move between the internal network and the outside, where our heavy forwarders are not accessible.

I attempted to do this by putting our heavy forwarders first in a comma-separated list in outputs.conf on the universal forwarder, and then followed it with the Splunk Cloud URLs. Port 9997 out to Splunk Cloud is only open from our heavy forwarders in the network, and our heavy forwarders are only accessible from inside the network, so in this way it seems like it should simply load balance to whichever server it can reach, but sending to Splunk Cloud requires special cert authentication. Because of this, I followed the instructions to enable the forwarder to work with our Splunk Cloud instance, which adds a couple of certs to an app on the forwarder. I suspect this will make it work fine to connect to Splunk Cloud directly, but I can see in the splunkd.log for the forwarder that when it tries to connect to one of our heavy forwarders, it fails, probably because it is trying to use the Splunk Cloud certs.

My questions are, is it possible to configure this so that the forwarder can send to either Splunk Cloud directly or a heavy forwarder, and is there some other, better way to handle this than what I'm thinking of? I know another solution would be to just skip the heavy forwarders and have these roaming devices send to Splunk Cloud directly regardless of where they are, but we'd like to avoid that unless that is the only option.

[gbeatty]

Is there a specific reason you want the data to route through heavy forwarders?  If it isn't because of fears of some sort of network straining, and you aren't doing parsing, it may make sense to just forward directly to Splunk Cloud.  That will make the load on your heavy forwarders a bit more predictable and may allow you to have fewer.  I know at one point you could only have one set of certs per instance of splunkd.