Deployment Architecture

Lookup too big to replicate but rarely changes

b_chris21
Communicator

Hello,

I have seen multiple posts related to large lookup files delaying the replication in a distributed environment.

In my case I have a lookup table of around 120MB that is used on an automatic lookup table so it has to be replicated to the search peers.

The lookup table file is static and rarely changed.

My questions are:

- Once the replication bundle syncs successfully, will Splunk SH try to replicate it again to peers if there is no change has been found?
- If file changes only by few lines/records, will Splunk try to replicate the delta from the previous state?

Bandwidth is limited so I don't want to have a bottleneck during operations.

Thank you in advance for your time.

With kind regards.

Chris

Labels (1)
Tags (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The lookup file should be sent only in a full bundle, which is sent when an indexer doesn't have a current one.  Otherwise, partial ("delta") bundles are sent containing only changes from the last full bundle.  I don't know just how "partial" a delta bundle is, however.

One option is to replicate the lookup file out-of-band (perhaps using rsync) and then blacklisting it from the replication bundle.

---
If this reply helps you, Karma would be appreciated.

b_chris21
Communicator

Thanks for your quick reply.

I will try to resync the file to indexers. 👌🏻 

Before opening this thread though, I manually copied the lookup file via scp to all indexers and blacklisted it in replication. Any search would give the error ""Could not load lookup=LOOKUP...".

Why did this trigger?

Thanks again. 

Best regards, 

Chris

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Make sure you put the lookups in the right places on the indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...