Because you've indicated that you're working within a Search Head Cluster (SHC), the options for this aren't exactly straightforward. The scheduled search you've built to trigger the rebuild of the lookup table is dispatched to one of the members of the SHC, not all of them. This is expected behavior, and coincides with what you're observing in your environment. An alternative to the "login by hand" option for triggering the search would be to remotely do so via REST / curl. I always refer back to this answer for triggering saved searches.
Another complication to this is that the lookup table is (typically) distributed down to the indexers, too, so that they can perform the enrichment in the "Map" part of "map reduce", meaning that they're doing work on the behalf of the search heads. This may also trigger that "create index file from large lookup" behavior on those hosts.
You might consider the KV store as an approach. The lookup abstraction will still play nice, and the KV store members will handle the replication for you.
However, since you've indicated that some of your lookups are GB-sized, I might cherry pick the fields that are most commonly used, build a lookup from that, and double-jump (lookup to key another lookup) in the worst case.
... View more