Deployment Architecture

how to use maxDataSize to control the size of each bucket

jek01
New Member

I use maxDataSize to control the size of each bucket.
The purpose is to limit each bucket up to 6MB.
Join "maxDataSize = 6 maxHotBuckets = 1" in test of local/indexes.conf.
(index=test)
But the result is a failure.
Execution results:
1. size of file=5MB ,Execution results:2 hot db,no any warm db. BUT Expected: only one bucket.
2. size of file=10MB ,Execution results:1 hot db,no any warm db. BUT Expected: only two buckets.

excuse me, can you tell me the reason? AND how can I achieve the goals?
TKS.

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

View solution in original post

jkat54
SplunkTrust
SplunkTrust

I believe your answer lies in this excerpt from indexes.conf:

rotatePeriodInSecs = <positive integer>
* Controls the service period (in seconds): how often splunkd performs
  certain housekeeping tasks.  Among these tasks are:
  * Check if a new hotdb needs to be created.
  * Check if there are any cold DBs that should be frozen.
  * Check whether buckets need to be moved out of hot and cold DBs, due to
    respective size constraints (i.e., homePath.maxDataSizeMB and
    coldPath.maxDataSizeMB)
* This value becomes the default value of the rotatePeriodInSecs attribute
  for all volumes (see rotatePeriodInSecs in the Volumes section)
* Highest legal value is 4294967295
* Defaults to 60.

Splunk is making its best effort to give you 6mb per bucket, but it's only checking to see if it needs to rotate buckets every minute. You could reduce this time period but i recommend a maxDataSize of at least 100. Why are you trying to make the buckets so small?

adonio
Ultra Champion

i think @jkat54 meant indexes.conf and not props.conf

jkat54
SplunkTrust
SplunkTrust

Thanks that's what I meant but I couldn't edit until just now because my account was locked.

0 Karma

jek01
New Member

Hi ... Thanks
Because I take a small data test, so every bucket is so small. I'll try again.

0 Karma
Get Updates on the Splunk Community!

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

Observability Newsletter Highlights | March 2023

 March 2023 | Check out the latest and greatestSplunk APM's New Tag Filter ExperienceSplunk APM has updated ...

Security Newsletter Updates | March 2023

 March 2023 | Check out the latest and greatestUnify Your Security Operations with Splunk Mission Control The ...