Deployment Architecture

how to use maxDataSize to control the size of each bucket

jek01
New Member

I use maxDataSize to control the size of each bucket.
The purpose is to limit each bucket up to 6MB.
Join "maxDataSize = 6 maxHotBuckets = 1" in test of local/indexes.conf.
(index=test)
But the result is a failure.
Execution results:
1. size of file=5MB ,Execution results:2 hot db,no any warm db. BUT Expected: only one bucket.
2. size of file=10MB ,Execution results:1 hot db,no any warm db. BUT Expected: only two buckets.

excuse me, can you tell me the reason? AND how can I achieve the goals?
TKS.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

View solution in original post

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

adonio
Ultra Champion

i think @jkat54 meant indexes.conf and not props.conf

jkat54
SplunkTrust
SplunkTrust

Thanks that's what I meant but I couldn't edit until just now because my account was locked.

0 Karma

jek01
New Member

Hi ... Thanks
Because I take a small data test, so every bucket is so small. I'll try again.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...