Deployment Architecture

Why do search head cluster members keep old bundle files, and can these be deleted safely?

att35
Builder

Hi,

We currently have a Search Head Cluster setup which has one deployer and two cluster members. One of the cluster members ran out of disk space and thus cannot issue searches anymore. Also, when I checked the cluster status, this one shows status as detention.

There are several bundle files under /opt/splunk/var/run, most of which are 1 GB +. The member which ran out of disk space is holding almost twice as many .bundle files under that folder as compared to the other member. Both were configured the same way and all apps were deployed only via Deployer, but how can there be such difference between them? Could these bundle files be something completely unrelated to SH Clustering?
Can any of these bundle files be deleted safely?

Also, around the same time one member had the disk issue, the other active member (which is also the captain now) had a replication failure for all the connected search peers. State is up and Health status is "Healthy", but Replication status is "Failed". Could this be related to the fact that the only other member is currently down?

Thanks,

~ Abhi

0 Karma
1 Solution

att35
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

View solution in original post

effem
Communicator

If you dont know how to actually get to know, whats space-consuming inside the bundle, then go to your searchhead:
tar -vtf <path to bundle> | awk '{print $3" "$4" "$5" "$6}' | sort -h
This prints the biggest files in the bundle on the bottom.

0 Karma

att35
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

att35
Builder

Update:

We changed maxBundleSize setting but it seems to be a temporary solution. Bundle size has now went up to 4-5GB. We added the blacklist stanza in distsearch.conf and removed some of the large lookup files but the bundle size is still the same.

Is there any way to find out why bundle is so big? If Splunk says 200MB+ is large then something must be seriously mis-configured for it to reach 5 GB. Both servers have ES installed with correlations enabled.

Kindly advise.

Thanks,
~Abhi

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...