Deployment Architecture

Knowledge Bundle Size Reduction

shivanshu1593
Builder

Hello All,

Recently, I observed error messages on my search head like "Unable to distribute to peer named XXX at URI https://xx:8089 because replication was unsuccessful. replicationStatus Failed failure info: Dispatch Command: Search Bundle throttling is occuring because the limit for number of bundles with pending lookups for indexing has been exceeded. This could be the result of large lookup files updating faster than Splunk software can index them. Throttling ends when this instance has caught up with indexing of lookups"

On investigating, I did found that a lot of our lookups were over 100+ MB, going upto 500+ MB were in the bundle. I proceeded to identify the large lookups and created a replicationBlacklist for them, which I plan to implement on my search head in distsearch.conf.

[replicationBlacklist]

blklistfiles = /apps/*/lookups/(abc.csv|def.csv|fgh.csv)

 

My question is, is it good to delete all the .bundle files from $SPLUNK_HOME/var/run directory, after implementing the above mentioned change and then restart Splunkd? Some bundles are almost a year old. What will be the impact of this, or is there anything I should take care of before doing this or is there an alternative?

Any opinion/advice will be highly welcomed.

Thank you,

S

 

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma

thambisetty
SplunkTrust
SplunkTrust

Splunk keeps last 5 bundles I believe, you are safe to remove more than 5 bundles.  you can delete all bundles but I don't recommend doing that. you will get so many inconsistent errors. once you have updated blacklist verify that lookups are blacklisted or not by just checking bundle size in var/run  if you see bundle size reduced wait for 10 mins max you should see new bundle pushed to search peers (indexers). 

 

————————————
If this helps, give a like below.
0 Karma

shivanshu1593
Builder

Hello @thambisetty,

Thanks for the guidance. Since there are chances of errors of inconsistency, if I do not remove any of the previous search bundles,  blacklist the lookups, and restart the service, will it cause Splunk to create a new knowledge bundle, or do I have to get rid of the previous knowledge bundle?

Also, do you think the way my blacklist is setup, will work?

Since it'll be done on a production environment, I don't want to cause any troubles 🙂

Thanks,

S

Thank you,
Shiv
###If you found the answer helpful, kindly consider upvoting/accepting it as the answer as it helps other Splunkers find the solutions to similar issues###
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...