Why do search head cluster members keep old bundle files, and can these be deleted safely?

[att35]

Hi,

We currently have a Search Head Cluster setup which has one deployer and two cluster members. One of the cluster members ran out of disk space and thus cannot issue searches anymore. Also, when I checked the cluster status, this one shows status as detention.

There are several bundle files under /opt/splunk/var/run, most of which are 1 GB +. The member which ran out of disk space is holding almost twice as many .bundle files under that folder as compared to the other member. Both were configured the same way and all apps were deployed only via Deployer, but how can there be such difference between them? Could these bundle files be something completely unrelated to SH Clustering?
Can any of these bundle files be deleted safely?

Also, around the same time one member had the disk issue, the other active member (which is also the captain now) had a replication failure for all the connected search peers. State is up and Health status is "Healthy", but Replication status is "Failed". Could this be related to the fact that the only other member is currently down?

Thanks,

~ Abhi

[effem]

If you dont know how to actually get to know, whats space-consuming inside the bundle, then go to your searchhead:
tar -vtf <path to bundle> | awk '{print $3" "$4" "$5" "$6}' | sort -h
This prints the biggest files in the bundle on the bottom.

[att35 · Accepted solution]

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

[att35]

Update:

We changed maxBundleSize setting but it seems to be a temporary solution. Bundle size has now went up to 4-5GB. We added the blacklist stanza in distsearch.conf and removed some of the large lookup files but the bundle size is still the same.

Is there any way to find out why bundle is so big? If Splunk says 200MB+ is large then something must be seriously mis-configured for it to reach 5 GB. Both servers have ES installed with correlations enabled.

Kindly advise.

Thanks,
~Abhi