We still have a bit of work to identify low level details but have solved this problem. It was solved by moving the scheduled search to a pool of search heads in another data center which are behind our API load balancer vip. Essentially my API request was going to a pool of servers in data center A and the scheduled search was running on a pool of servers in data center B. It worked without issue when we moved the scheduled search to data center A.
Thanks to jkat54 to throwing out ideas.
... View more