topic Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf? in Installation

How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tokio13 — Thu, 17 Feb 2022 17:51:39 GMT

Hello,

I'm experiencing the following issue on one of my search heads (total of 3):

Knowledge bundle size=2608MB exceeds max limit=2000MB. Distributed searches are running against an outdated knowledge bundle. Please remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf.

Why is the SH behaving like this when the others have the same config?

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

isoutamo — Thu, 17 Feb 2022 18:00:38 GMT

Probably you have big lookups on your sh which it try to send to IDX layer? If you have SHC then captain is that node which try to send those search bundles to IDXs. You could check this from MC (Search - distributed search).

You could exclude those lookups by size or name from bundle. Then you must use in lookup command that it will be executed on sh layer when you are using those lookups.

You could found many questions an answers from community about that issue.
Like this https://community.splunk.com/t5/Splunk-Search/Large-lookup-caused-the-bundle-replication-to-fail-What-are-my/m-p/194594

r. Ismo

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

gcusello — Fri, 18 Feb 2022 07:21:16 GMT

Hi @tokio13,

as @isoutamo said, probably you have very large lookups that are sent from SHs to the Indexers.

As you can see at https://docs.splunk.com/Documentation/Splunk/8.2.4/Admin/Distsearchconf , you have to blacklist some (or all) of them in distsearch.conf

[replicationBlacklist] blacklist1 = lookup1 blacklist2 = lookup2

Ciao.

Giuseppe

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tokio13 — Thu, 24 Feb 2022 15:47:19 GMT

Unfortunately I was unable to resolve my issue with the mentioned answers. I'm still working on this but I appreciate you suggestions.

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

gcusello — Thu, 24 Feb 2022 16:07:19 GMT

Hi @tokio13,

what's the problem: you cannot access distsearch.conf or what else?

Could you share more infos?

I already solved the same problem in one of our customers.

Ciao.

Giuseppe

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

somesoni2 — Thu, 24 Feb 2022 16:25:50 GMT

Are you using search head clustering for those 3 SH of yours? It could be a local artifact on that SH which is causing knowledge bundle to be large from that SH.

Using instructions from below to check the details about knowledge bundle in troubling SH and compare it with SH that is not having this issue.

https://docs.splunk.com/Documentation/Splunk/8.2.4/DistSearch/Troubleshootknowledgebundlereplication#Use_the_CLI_to_view_bundle_replication_configuration_and_status

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tokio13 — Thu, 24 Feb 2022 16:39:06 GMT

I have access to distsearch.conf on all of my search heads (3) .

In this environment Cluster Master instance acts as Deployer. And the deployer acts like CM, they sit on the same instance. (+3IDX)

I get the [ Knowledge bundle size=2608MB exceeds max limit=2000MB. Distributed searches are running against an outdated knowledge bundle. Please remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf ] notification on the capitan of the search head cluster.

This affects my searches:

Expected common latest bundle version on all peers after sync replication, found none. Reverting to old behavior - using most recent bundles on all

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tokio13 — Thu, 24 Feb 2022 17:34:25 GMT

I followed the documentation that you mentioned and everything looks the same on all my three Search Heads.

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tshah-splunk — Fri, 25 Feb 2022 04:31:08 GMT

Hey @tokio13,

Can you run the btool command to check what value is configured for the maxBundleSize parameter on one of your SHC members?

$SPLUNK_HOME/bin/splunk btool distsearch list --debug | grep maxBundleSize

If this returns you value less than 2000, consider having the value of the parameter updated to a higher value. than the limit.

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

gcusello — Fri, 25 Feb 2022 06:33:49 GMT

Hi @tokio13 ,

if you can access the distsearch.conf, why cannot you use my solution?

I used it few days ago in a project with Splunk Professional Services.

Ciao.

Giuseppe

Re: How to remove/disable files from knowledge bundle or increase maxBundleSize in distsearch.conf?

tokio13 — Mon, 28 Feb 2022 12:54:44 GMT

I was able to solve the problem by going to the /opt/splunk/etc/apps/search/lookups/

and removing a .csv that was exported by an old search query (output) and was actually not needed.

Thanks every one once again!