Deployment Architecture

how to roll hot bucket to warm at specific time intervals?

gizemk00
Engager

I want to change time of buckets transitions
from hot to warm or warm to cold etc.

Tags (1)
0 Karma
1 Solution

adonio
Ultra Champion

hello there,
look at this configuration in indexes.conf

maxHotSpanSecs = <positive integer>
* Upper bound of timespan of hot/warm buckets in seconds.
* NOTE: If you set this too small, you can get an explosion of hot/warm
  buckets in the filesystem.
* NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all
  events to the single hot bucket and maxHotSpanSeconds will not be
  enforced.
* If you set this parameter to less than 3600, it will be automatically
  reset to 3600.
* This is an advanced parameter that should be set
  with care and understanding of the characteristics of your data.
* Highest legal value is 4294967295
* Defaults to 7776000 seconds (90 days).
* Note that this limit will be applied per ingestion pipeline. For more
  information about multiple ingestion pipelines see parallelIngestionPipelines
  in server.conf.spec file.
* With N parallel ingestion pipelines, each ingestion pipeline will write to
  and manage its own set of hot buckets, without taking into account the state
  of hot buckets managed by other ingestion pipelines.  Each ingestion pipeline
  will independently apply this setting only to its own set of hot buckets.
* NOTE: the bucket timespan snapping behavior is removed from this setting. 
  See the 6.5 spec file for details of this behavior.

note, you will probably want to adjust other settings as well, for example, the max size of a bucket "maxDataSize" and also maybe the maximum hot buckets and maximum warm buckets. you will probably will have more considerations as each index (most of the time) grows in a different paste / pattern.
also, pay attention to the comment: "This is an advanced parameter that should be set with care and understanding of the characteristics of your data"

hope it helps

View solution in original post

adonio
Ultra Champion

hello there,
look at this configuration in indexes.conf

maxHotSpanSecs = <positive integer>
* Upper bound of timespan of hot/warm buckets in seconds.
* NOTE: If you set this too small, you can get an explosion of hot/warm
  buckets in the filesystem.
* NOTE: If you set maxHotBuckets to 1, Splunk attempts to send all
  events to the single hot bucket and maxHotSpanSeconds will not be
  enforced.
* If you set this parameter to less than 3600, it will be automatically
  reset to 3600.
* This is an advanced parameter that should be set
  with care and understanding of the characteristics of your data.
* Highest legal value is 4294967295
* Defaults to 7776000 seconds (90 days).
* Note that this limit will be applied per ingestion pipeline. For more
  information about multiple ingestion pipelines see parallelIngestionPipelines
  in server.conf.spec file.
* With N parallel ingestion pipelines, each ingestion pipeline will write to
  and manage its own set of hot buckets, without taking into account the state
  of hot buckets managed by other ingestion pipelines.  Each ingestion pipeline
  will independently apply this setting only to its own set of hot buckets.
* NOTE: the bucket timespan snapping behavior is removed from this setting. 
  See the 6.5 spec file for details of this behavior.

note, you will probably want to adjust other settings as well, for example, the max size of a bucket "maxDataSize" and also maybe the maximum hot buckets and maximum warm buckets. you will probably will have more considerations as each index (most of the time) grows in a different paste / pattern.
also, pay attention to the comment: "This is an advanced parameter that should be set with care and understanding of the characteristics of your data"

hope it helps

gizemk00
Engager

This is the answer that I'm accepting. thank you

0 Karma

ddrillic
Ultra Champion

@gizemk00, just please be careful with maxHotSpanSecs, with a low value and a slow growing index, you can produce too many buckets, which is not recommended.

0 Karma

bic
Explorer

You can set

maxHotSpanSecs =

https://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/Indexesconf#GLOBAL_SETTINGS

You also will have to set similar for hot to cold.

0 Karma
Get Updates on the Splunk Community!

How to Monitor Google Kubernetes Engine (GKE)

We’ve looked at how to integrate Kubernetes environments with Splunk Observability Cloud, but what about ...

Index This | How can you make 45 using only 4?

October 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Splunk Education Goes to Washington | Splunk GovSummit 2024

If you’re in the Washington, D.C. area, this is your opportunity to take your career and Splunk skills to the ...