Why does merge-buckets only merge up to 300 buckets?

schose — Mon, 23 May 2022 15:20:21 GMT

Hi all,

I'm checking out the "merge-buckets" command. I created an index with 1000 events per bucket. in sum my index have

~/splunk/bin/splunk search "| dbinspect index=testbuckets2 | stats count" count ----- 5479

buckets.

~/splunk/bin/splunk merge-buckets --index-name=testbuckets2 --min-size=1 --max-count=1000 Using the following config: --max-count=1000 --min-size=1 --max-size=1000 --max-timespan=7776000 Found (300) buckets to merge. Starting to merge (300) buckets. Number of buckets already merged: 0/300 (0.00%). New Bucket: /Users/andreas/splunk/var/lib/splunk/testbuckets2/db/db_1653310364_1653310268_17359 Number of buckets merged: 300/300 (100.00%). Number of buckets created: 1. Time taken: 27 seconds, 21 milliseconds

after the operation i see 299 buckets less

~/splunk/bin/splunk search "| dbinspect index=testbuckets2 | stats count" count ----- 5180

running merge-bucket a second time doesn't merge any further buckets. It seems there is a hardcoded limit of 300 buckets?! any good reason for this?

best regards,

Andreas

Re: Why does merge-buckets only merge up to 300 buckets?

hytan — Mon, 19 Sep 2022 00:56:58 GMT

Apologies for seeing this post so late. Hopefully the response can still be of help to others.

There is a default bucket count for merging, but that is overridden by the --max-count=1000 parameter specified.

There could be various reasons for not merging more buckets:

1) The time span for a bucket is defaulted to 7776000secs (90 days). If the buckets are very spread out, the buckets may not be large enough to be merged to meet the min-size of 1MB that was specified (--min-size=1)

2) Only warm buckets can be merged. Cold, hot and frozen buckets cannot be merged.

The count of 300 is most likely due to this parameter maxWarmDBCount. The default is only 300 warm buckets. Once exceeded, Splunk will roll the warm buckets to cold, and you will not be able to merge them. For your test, you can change this to a really large number.

maxWarmDBCount = <nonnegative integer>
* The maximum number of warm buckets.
* Warm buckets are located in the 'homePath' for the index.
* If set to zero, splunkd does not retain any warm buckets
  It rolls the buckets to cold as soon as it is able.
* Splunkd ignores this setting on remote storage enabled indexes.
* Highest legal value is 4294967295.
* Default: 300

This is the link to the official docs.

https://docs.splunk.com/Documentation/Splunk/9.0.1/Troubleshooting/CommandlinetoolsforusewithSupport

I would recommend also adding the following parameters --dryrun and --debug.

~/splunk/bin/splunk merge-buckets --index-name=testbuckets2 --min-size=1 --max-count=1000 --dryrun --debug

topic Why does merge-buckets only merge up to 300 buckets? in Splunk Enterprise

Why does merge-buckets only merge up to 300 buckets?

Re: Why does merge-buckets only merge up to 300 buckets?