Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

[dmenon84]

Hi,

I see the following error on one of my search heads since yesterday. Tried different things but haven't been able to fix it yet. Thanks in advance.

Error message:

Problem replicating config (bundle) to search peer '1.2.3.4:8089', error while transmitting bundle data.

splunkd.log

05-16-2016 14:31:01.994 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer1.com at uri https://1.2.3.4:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:01.995 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer2.com at uri https://5.5.6.7:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:01.995 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer3.com at uri https://9.0.8.9:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:02.465 -0500 INFO Archiver - Archiving large_file=/opt/splunk/etc/apps/app/lookups/identity.csv of size_in_bytes=265366446 (exceeding concerning_threshold=52428800)
05-16-2016 14:31:06.068 -0500 INFO Archiver - Archiving large_file=/opt/splunk/etc/apps/app/lookups/email_activity.csv of size_in_bytes=418267845 (exceeding concerning_threshold=52428800)

I added these to the distsearch.conf under etc/system/local

[replicationSettings]
sendRcvTimeout = 120

[replicationWhitelist]
allConf =*.conf

[replicationBlacklist]
identity_lookup = *identity.csv
email_activity_lookup = *email_activity.csv

and did a splunk restart, but that did not fix my issue.

The lookups are in a custom app in Search Head 2 which is down, but search head 1 is working fine.

[highsplunker]

I think here's the solution (by rphillips_splunk)
https://answers.splunk.com/answers/85868/knowledge-bundle.html

[ppablo]

Hi @dmenon84

You just tried posting an exact duplicate of this question just now. Please do not do that. Comment on @jkat54's answer below with more information if it didn't fully answer your question.

[jkat54 · Accepted solution]

Try this instead:

[replicationBlacklist]
identity_lookup = ...identity.csv
email_activity_lookup = ...email_activity.csv

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf <- says the following:
* The whitelist_pattern is the Splunk-style pattern matching, which is primarily
regex-based with special local behavior for '...' and '*'.
* ... matches anything, while * matches anything besides directory separators.
See props.conf.spec for more detail on these.
* Note '.' will match a literal dot, not any character.`

[dmenon84]

This is strange, when I posted the question the first time I never got a confirmation that I was successfully able to post. I thought it was a browser issue so i tried from a different browser and again the same thing happened. I was under impression that I was not able to post my question at all. Today when I logged in, I found out that my question was posted and just saw your answer. I solved this issue last week doing exactly what you suggested. I tried blacklisting on the master node and then restarted the Master but the issue remained so I blacklisted the lookups on the Search Head that was down and that fixed it. Can you please explain me why the lookups have to be blacklisted and if there is any other steps that we can take to avoid this replication issue in future without having to blacklist the lookup every time?

[jkat54]

It's just how the splunk application works.

There is another option where you can put the lookups in a shared location and instruct the indexers and search heads to use that shared location. If done incorrectly however (bad nfs etc) it can drastically affect performance.

Also another option is to force your lookup to only be used on the instance you dispatched it from by adding local=true to your lookup command as mentioned in this doc: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Lookup

[dmenon84]

my distsearch.conf looks like this
[replicationBlacklist]
identity_lookup = *identity.csv
email_activity_lookup = *email_activity.csv