Deployment Architecture

Why do search head cluster members keep old bundle files, and can these be deleted safely?

abhijittikekar
Builder

Hi,

We currently have a Search Head Cluster setup which has one deployer and two cluster members. One of the cluster members ran out of disk space and thus cannot issue searches anymore. Also, when I checked the cluster status, this one shows status as detention.

There are several bundle files under /opt/splunk/var/run, most of which are 1 GB +. The member which ran out of disk space is holding almost twice as many .bundle files under that folder as compared to the other member. Both were configured the same way and all apps were deployed only via Deployer, but how can there be such difference between them? Could these bundle files be something completely unrelated to SH Clustering?
Can any of these bundle files be deleted safely?

Also, around the same time one member had the disk issue, the other active member (which is also the captain now) had a replication failure for all the connected search peers. State is up and Health status is "Healthy", but Replication status is "Failed". Could this be related to the fact that the only other member is currently down?

Thanks,

~ Abhi

0 Karma
1 Solution

abhijittikekar
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

View solution in original post

effem
Communicator

If you dont know how to actually get to know, whats space-consuming inside the bundle, then go to your searchhead:
tar -vtf <path to bundle> | awk '{print $3" "$4" "$5" "$6}' | sort -h
This prints the biggest files in the bundle on the bottom.

0 Karma

abhijittikekar
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

View solution in original post

abhijittikekar
Builder

Update:

We changed maxBundleSize setting but it seems to be a temporary solution. Bundle size has now went up to 4-5GB. We added the blacklist stanza in distsearch.conf and removed some of the large lookup files but the bundle size is still the same.

Is there any way to find out why bundle is so big? If Splunk says 200MB+ is large then something must be seriously mis-configured for it to reach 5 GB. Both servers have ES installed with correlations enabled.

Kindly advise.

Thanks,
~Abhi

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!