Deployment Architecture

Lookup too big to replicate but rarely changes

b_chris21
Communicator

Hello,

I have seen multiple posts related to large lookup files delaying the replication in a distributed environment.

In my case I have a lookup table of around 120MB that is used on an automatic lookup table so it has to be replicated to the search peers.

The lookup table file is static and rarely changed.

My questions are:

- Once the replication bundle syncs successfully, will Splunk SH try to replicate it again to peers if there is no change has been found?
- If file changes only by few lines/records, will Splunk try to replicate the delta from the previous state?

Bandwidth is limited so I don't want to have a bottleneck during operations.

Thank you in advance for your time.

With kind regards.

Chris

Labels (1)
Tags (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

b_chris21
Communicator

Thanks for your quick reply.

I will try to resync the file to indexers. 👌🏻 

Before opening this thread though, I manually copied the lookup file via scp to all indexers and blacklisted it in replication. Any search would give the error ""Could not load lookup=LOOKUP...".

Why did this trigger?

Thanks again. 

Best regards, 

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure you put the lookups in the right places on the indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...