topic Why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk? in Deployment Architecture

Why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

richarddicaire — Thu, 31 Mar 2022 13:16:16 GMT

Splunk 6.6.3, clustered env. One of our indexers reporting high disk usage. Traced it down to /opt/splunk/var/run/splunk/cluster/search-buckets containing many search_sitedefault_gen*.csv.gz and summarize_sitedefault_gen*.csv.gz files going back to 22 days ago (December 12 at this time). I deleted older ones to stop triggering our disk use alerts.

Whats creating these files and why?

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

richarddicaire — Fri, 05 Jan 2018 15:33:46 GMT

We're now seeing additional indexers having disk usage issues from the above, can anyone shed any light?

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

ddrillic — Fri, 05 Jan 2018 16:35:42 GMT

We had recently a similar but different path issue at Why does /opt/splunk/var/run/searchpeers fill up?

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

richarddicaire — Fri, 05 Jan 2018 16:44:58 GMT

@ddrillic thanks for responding but not related. I need to know what is creating the above files in /opt/splunk/var/run/splunk/cluster/search-buckets. I just had to delete files from all of my indexers to have available space. Never had to do this before our upgrade to 6.6.3.

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

richarddicaire — Mon, 08 Jan 2018 16:40:09 GMT

Hi, can anyone provide input as to what is creating search_sitedefault_gen*.csv.gz and summarize_sitedefault_gen*.csv.gz files in /opt/splunk/var/run/splunk/cluster/search-buckets?

Thanks

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

masonmorales — Mon, 25 Jun 2018 21:34:59 GMT

Same issue here on v6.6.5. Did you ever find anything out?

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

davpx — Sat, 26 Jan 2019 03:12:51 GMT

You have a lot of traffic for your deployment. Increase disk space.

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

BainM — Mon, 25 Feb 2019 15:30:26 GMT

This is NOT a helpful answer and does not explain why there are so many of these files in this directory path. There apparently is no documentation from Splunk on this. I am opening a case as I suggest everyone else having this does the same.

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

dxu_splunk — Tue, 29 Sep 2020 23:33:21 GMT

This was a combination of two bugs that were fixed in later versions of splunk (7.0.8+, 7.1.6+, 7.2.4+)

For a workaround, its safe to

delete older generation files, keeping the last 10 or so per site
don't delete the gen0 file

for example, if i have:
search_sitedefault_gen1000.csv.gz as the latest file, i can delete search_sitedefault_gen(1-990).csv.gz safely

but remember this is per site, so if i have the latest:

search_site0_gen1000.csv.gz (delete gen1-990 for site0, dont delete gen0)
search_site1_gen3500.csv.gz (delete gen1-3490 for site1, dont delete gen0)

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

santu27487kanna — Thu, 10 Oct 2019 05:16:04 GMT

Hi dxu,

Is there a workaround for the same?

Thanks,
Santhosh

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

stepheneardley — Tue, 05 Nov 2019 10:35:18 GMT

FWIW and I know it's not ideal but a rolling restart of the cluster peers will clear these down. I'm on 7.1.5.

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

santu27487kanna — Tue, 05 Nov 2019 10:43:00 GMT

Thank you stepheneardley.

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

jk01571 — Tue, 07 Jan 2020 23:56:41 GMT

What is the purpose of the file?
And do you know if there is a cycle or setting method to delete automatically?

Re: why is /opt/splunk/var/run/splunk/cluster/search-buckets filling up my disk?

Sahr_Lebbie — Thu, 31 Mar 2022 01:33:40 GMT

Anyone else facing same issues in 8.2.4. Will check with support and see.