Is there anything special about interacting with a Search Head Cluster via the REST APIs? Specifically, what endpoint should you hit? Can splunkd (port 8089) be placed behind a load balancer of some kind? SplunkWeb provides some failover and transparent result proxying to minimize user impact, but it's unclear to me how much of that happens within splunkd and how much of the magic is in SplunkWeb.
I assume the easiest approach is just to point the Python SDK to a single SHC member and hope for the best, but I'm guessing there's a better way to do it.
Search head clusters will insure the search completes regardless of if you use web or api. In fact every command line and every web button has a corresponding api call.
So you can literally dispatch a search via api on 8089 and check back when it's done and get the results. It won't matter which search head API you hit. So yeah I'd put it behind a load balancer and then maybe use a low length of time for persistence or maybe persist based on source ip.