Lookup too big to replicate but rarely changes

[b_chris21]

Hello,

I have seen multiple posts related to large lookup files delaying the replication in a distributed environment.

In my case I have a lookup table of around 120MB that is used on an automatic lookup table so it has to be replicated to the search peers.

The lookup table file is static and rarely changed.

My questions are:

- Once the replication bundle syncs successfully, will Splunk SH try to replicate it again to peers if there is no change has been found?
- If file changes only by few lines/records, will Splunk try to replicate the delta from the previous state?

Bandwidth is limited so I don't want to have a bottleneck during operations.

Thank you in advance for your time.

With kind regards.

Chris

[richgalloway · Accepted solution]

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

[b_chris21]

Thanks for your quick reply.

I will try to resync the file to indexers. 👌🏻 

Before opening this thread though, I manually copied the lookup file via scp to all indexers and blacklisted it in replication. Any search would give the error ""Could not load lookup=LOOKUP...".

Why did this trigger?

Thanks again. 

Best regards, 

Chris

[richgalloway]

Make sure you put the lookups in the right places on the indexers.