Deployment Architecture

Why do search head cluster members keep old bundle files, and can these be deleted safely?

att35
Builder

Hi,

We currently have a Search Head Cluster setup which has one deployer and two cluster members. One of the cluster members ran out of disk space and thus cannot issue searches anymore. Also, when I checked the cluster status, this one shows status as detention.

There are several bundle files under /opt/splunk/var/run, most of which are 1 GB +. The member which ran out of disk space is holding almost twice as many .bundle files under that folder as compared to the other member. Both were configured the same way and all apps were deployed only via Deployer, but how can there be such difference between them? Could these bundle files be something completely unrelated to SH Clustering?
Can any of these bundle files be deleted safely?

Also, around the same time one member had the disk issue, the other active member (which is also the captain now) had a replication failure for all the connected search peers. State is up and Health status is "Healthy", but Replication status is "Failed". Could this be related to the fact that the only other member is currently down?

Thanks,

~ Abhi

0 Karma
1 Solution

att35
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

View solution in original post

effem
Communicator

If you dont know how to actually get to know, whats space-consuming inside the bundle, then go to your searchhead:
tar -vtf <path to bundle> | awk '{print $3" "$4" "$5" "$6}' | sort -h
This prints the biggest files in the bundle on the bottom.

0 Karma

att35
Builder

We found out that this large bundle was mainly due to two files from DSA app which were quite big in size(both csv lookup files). These files were removed from bundle which also resolved bundle replication issues.

Thanks,

~ Abhi

att35
Builder

Update:

We changed maxBundleSize setting but it seems to be a temporary solution. Bundle size has now went up to 4-5GB. We added the blacklist stanza in distsearch.conf and removed some of the large lookup files but the bundle size is still the same.

Is there any way to find out why bundle is so big? If Splunk says 200MB+ is large then something must be seriously mis-configured for it to reach 5 GB. Both servers have ES installed with correlations enabled.

Kindly advise.

Thanks,
~Abhi

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...