Deployment Architecture

how to use maxDataSize to control the size of each bucket

jek01
New Member

I use maxDataSize to control the size of each bucket.
The purpose is to limit each bucket up to 6MB.
Join "maxDataSize = 6 maxHotBuckets = 1" in test of local/indexes.conf.
(index=test)
But the result is a failure.
Execution results:
1. size of file=5MB ,Execution results:2 hot db,no any warm db. BUT Expected: only one bucket.
2. size of file=10MB ,Execution results:1 hot db,no any warm db. BUT Expected: only two buckets.

excuse me, can you tell me the reason? AND how can I achieve the goals?
TKS.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

View solution in original post

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

adonio
Ultra Champion

i think @jkat54 meant indexes.conf and not props.conf

jkat54
SplunkTrust
SplunkTrust

Thanks that's what I meant but I couldn't edit until just now because my account was locked.

0 Karma

jek01
New Member

Hi ... Thanks
Because I take a small data test, so every bucket is so small. I'll try again.

0 Karma
Get Updates on the Splunk Community!

Dashboard Studio Challenge - Learn New Tricks, Showcase Your Skills, and Win Prizes!

Reimagine what you can do with your dashboards. Dashboard Studio is Splunk’s newest dashboard builder to ...

Introducing Edge Processor: Next Gen Data Transformation

We get it - not only can it take a lot of time, money and resources to get data into Splunk, but it also takes ...

Take the 2021 Splunk Career Survey for $50 in Amazon Cash

Help us learn about how Splunk has impacted your career by taking the 2021 Splunk Career Survey. Last year’s ...