Splunk Enterprise

Why does merge-buckets only merge up to 300 buckets?

schose
Builder

Hi all,

I'm checking out the "merge-buckets" command. I created an index with 1000 events per bucket. in sum my index have 

 

~/splunk/bin/splunk search "| dbinspect index=testbuckets2 | stats count"
count
-----
5479

 

 

buckets.

 

~/splunk/bin/splunk merge-buckets --index-name=testbuckets2 --min-size=1 --max-count=1000 
Using the following config: --max-count=1000 --min-size=1 --max-size=1000 --max-timespan=7776000
Found (300) buckets to merge.

Starting to merge (300) buckets. Number of buckets already merged: 0/300 (0.00%).
New Bucket:
/Users/andreas/splunk/var/lib/splunk/testbuckets2/db/db_1653310364_1653310268_17359

Number of buckets merged: 300/300 (100.00%).
Number of buckets created: 1.
Time taken: 27 seconds, 21 milliseconds

 

 

after the operation i see 299 buckets less

 

~/splunk/bin/splunk search "| dbinspect index=testbuckets2 | stats count"

count
-----
5180

 

 

running merge-bucket a second time doesn't merge any further buckets.  It seems there is a hardcoded limit of 300 buckets?! any good reason for this?

best regards,

Andreas

Labels (2)
Tags (1)
0 Karma

hytan
Splunk Employee
Splunk Employee

Apologies for seeing this post so late. Hopefully the response can still be of help to others.

There is a default bucket count for merging, but that is overridden by the --max-count=1000 parameter specified.

There could be various reasons for not merging more buckets:

1) The time span for a bucket is defaulted to 7776000secs (90 days). If the buckets are very spread out, the buckets may not be large enough to be merged to meet the min-size of 1MB that was specified (--min-size=1)

2) Only warm buckets can be merged. Cold, hot and frozen buckets cannot be merged.

The count of 300 is most likely due to this parameter maxWarmDBCount. The default is only 300 warm buckets. Once exceeded, Splunk will roll the warm buckets to cold, and you will not be able to merge them. For your test, you can change this to a really large number.

maxWarmDBCount = <nonnegative integer>
* The maximum number of warm buckets.
* Warm buckets are located in the 'homePath' for the index.
* If set to zero, splunkd does not retain any warm buckets
  It rolls the buckets to cold as soon as it is able.
* Splunkd ignores this setting on remote storage enabled indexes.
* Highest legal value is 4294967295.
* Default: 300

This is the link to the official docs.

https://docs.splunk.com/Documentation/Splunk/9.0.1/Troubleshooting/CommandlinetoolsforusewithSupport

I would recommend also adding the following parameters --dryrun and --debug.

 

~/splunk/bin/splunk merge-buckets --index-name=testbuckets2 --min-size=1 --max-count=1000 --dryrun --debug

 

 

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...