Deployment Architecture

Lookup too big to replicate but rarely changes

b_chris21
Communicator

Hello,

I have seen multiple posts related to large lookup files delaying the replication in a distributed environment.

In my case I have a lookup table of around 120MB that is used on an automatic lookup table so it has to be replicated to the search peers.

The lookup table file is static and rarely changed.

My questions are:

- Once the replication bundle syncs successfully, will Splunk SH try to replicate it again to peers if there is no change has been found?
- If file changes only by few lines/records, will Splunk try to replicate the delta from the previous state?

Bandwidth is limited so I don't want to have a bottleneck during operations.

Thank you in advance for your time.

With kind regards.

Chris

Labels (1)
Tags (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

b_chris21
Communicator

Thanks for your quick reply.

I will try to resync the file to indexers. 👌🏻 

Before opening this thread though, I manually copied the lookup file via scp to all indexers and blacklisted it in replication. Any search would give the error ""Could not load lookup=LOOKUP...".

Why did this trigger?

Thanks again. 

Best regards, 

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure you put the lookups in the right places on the indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...