Splunk Search
Highlighted

Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

Path Finder

Hi,

I see the following error on one of my search heads since yesterday. Tried different things but haven't been able to fix it yet. Thanks in advance.

Error message:

Problem replicating config (bundle) to search peer '1.2.3.4:8089', error while transmitting bundle data.

splunkd.log

05-16-2016 14:31:01.994 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer1.com at uri https://1.2.3.4:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:01.995 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer2.com at uri https://5.5.6.7:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:01.995 -0500 WARN DistributedPeerManager - Unable to distribute to peer named indexer3.com at uri https://9.0.8.9:8089 because replication was unsuccessful. replicationStatus Failed
05-16-2016 14:31:02.465 -0500 INFO Archiver - Archiving large_file=/opt/splunk/etc/apps/app/lookups/identity.csv of size_in_bytes=265366446 (exceeding concerning_threshold=52428800)
05-16-2016 14:31:06.068 -0500 INFO Archiver - Archiving large_file=/opt/splunk/etc/apps/app/lookups/email_activity.csv of size_in_bytes=418267845 (exceeding concerning_threshold=52428800)

I added these to the distsearch.conf under etc/system/local

[replicationSettings]
sendRcvTimeout = 120

[replicationWhitelist]
allConf =*.conf

[replicationBlacklist]
identity_lookup = *identity.csv
email_activity_lookup = *email_activity.csv

and did a splunk restart, but that did not fix my issue.

The lookups are in a custom app in Search Head 2 which is down, but search head 1 is working fine.

Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

SplunkTrust
SplunkTrust

Try this instead:

[replicationBlacklist]
identitylookup = ...identity.csv
email
activitylookup = ...emailactivity.csv

http://docs.splunk.com/Documentation/Splunk/6.0/admin/Distsearchconf <- says the following:
* The whitelist_pattern is the Splunk-style pattern matching, which is primarily
regex-based with special local behavior for '...' and '*'.
* ... matches anything, while * matches anything besides directory separators.
See props.conf.spec for more detail on these.
* Note '.' will match a literal dot, not any character.`

View solution in original post

Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

Path Finder

This is strange, when I posted the question the first time I never got a confirmation that I was successfully able to post. I thought it was a browser issue so i tried from a different browser and again the same thing happened. I was under impression that I was not able to post my question at all. Today when I logged in, I found out that my question was posted and just saw your answer. I solved this issue last week doing exactly what you suggested. I tried blacklisting on the master node and then restarted the Master but the issue remained so I blacklisted the lookups on the Search Head that was down and that fixed it. Can you please explain me why the lookups have to be blacklisted and if there is any other steps that we can take to avoid this replication issue in future without having to blacklist the lookup every time?

0 Karma
Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

Path Finder

my distsearch.conf looks like this
[replicationBlacklist]
identitylookup = *identity.csv
email
activitylookup = *emailactivity.csv

0 Karma
Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

SplunkTrust
SplunkTrust

It's just how the splunk application works.

There is another option where you can put the lookups in a shared location and instruct the indexers and search heads to use that shared location. If done incorrectly however (bad nfs etc) it can drastically affect performance.

Also another option is to force your lookup to only be used on the instance you dispatched it from by adding local=true to your lookup command as mentioned in this doc: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Lookup

0 Karma
Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

Community Manager
Community Manager

Hi @dmenon84

You just tried posting an exact duplicate of this question just now. Please do not do that. Comment on @jkat54's answer below with more information if it didn't fully answer your question.

0 Karma
Highlighted

Re: Why am I getting error "problem replicating config (bundle) to search peer..." on one search head and am getting no search results?

Contributor

I think here's the solution (by rphillips_splunk)
https://answers.splunk.com/answers/85868/knowledge-bundle.html

0 Karma