Bucket Resizing

[silverKi]

My friend and I have the same indexes.conf, but why are the bucket sizes being created different? Mine is around 1MB, but my friend's are created in 5.x MB units..

indexes.conf
[volume:hot]
path = /data/HOT
maxVolumeDataSizeMB = 100
 
[volume:cold]
path = /data/COLD
maxVolumeDataSizeMB = 100
 
[lotte]
homePath = volume:hot/lotte/db
coldPath = volume:cold/lotte/colddb
maxDataSize = 1
maxTotalDataSizeMB = 200
thawedPath = $SPLUNK_DB/lotte/thaweddb

[tscroggins]

Hi @silverKi,

The maxDataSize for your hot buckets is 1 MB. Your friend's setting appears to be higher (5 MB).

To add to what's already been written, you're writing (compressed) data at different rates:

Friend: ~720 bytes per second
You: ~19 bytes per second

This will influence the size of the warm bucket after it rolls from hot when either maxDataSize (1 MB in your case) or the default maxHotSpanSecs value of 90 days has been exceeded.

Hot buckets can also roll to warm when Splunk is restarted or when triggered manually. That probably isn't happening here, but it's worth noting.

[PickleRick]

You have very little data in your buckets. And comparing bucket sizes from two different environments with different data (especially if there's so little of that data) makes no sense.

Normally you'd expect buckets of several dozens or even hundreds of megabytes.

[isoutamo]

Hi

there are several reasons which can cause to switch a new bucket event it's max size is reached. 

When you are looking how your configuration has done. you should always use btool instead of looking those from file. Btool tolds you how splunk see those configurations as usually those are combined from several files.

You both should use 

splunk btool indexes list --debug lotte

to see what is actual configuration for index lotte. 

One reason for small bucket can be source events which contains events which have time stamps from past and future. Basically those haven't continuous increasing timestamps.

When I look those smaller buckets there seem to be this kind of behavior based on those epoch times in bucket names.

r. Ismo

[silverKi]

My configuration has not changed. 
I have verified that buckets are being created, and I have verified that a hot_quar_v1 bucket is being created. Why is it being created and how do I remove it?