Deployment Architecture

Best practices for hot/warm bucket retention?

splunker969
Communicator

The primary indexers data (Hot+ Warm) data is being full .Please help us in solving this issues . .We are trying to shrink the hot and warm are our primary indexers .
The retention period for hot +warm is 30 days.

What are best practice process? To come out of this issue .

somesoni2
Revered Legend

Data in hot/warm buckets are not managed by Splunk's data retention policy. Data is only frozen (archived or deleted based on what you've configured) from cold bucket directory. If you want to reduce the amount of data you want to retain for your index, ensure that you've configuration to have warm buckets rolled to cold buckets. See this for list of attributes that affect the data buckets:
http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Configureindexstorage#Attributes_that_affe...

Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.

0 Karma

splunker969
Communicator

Thanks @somesoni2

0 Karma

Sukisen1981
Champion

Correctly explained in detail by @somesoni2 . The data retention policy of 30 days is not a guarantee that ALL data in a 30 day period will be stored in the hot / wam buckets. What is important is to know how much data you are indexing. Things can get a bit complex here. Your organization will pay Splunk for the maximum amount of data indexed in a 24 hour period, but since you are not receiving any license violation warnings, I assume that is not a concern at this point of time.
@somesoni2 says - Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.
You need to make a decision on what size (how much GB) of your data is needed in hot and warm buckets.

0 Karma

splunker969
Communicator

Thanks @sukisen1981

0 Karma

koshyk
Super Champion

I'm not sure what you meant by Shrink? That's genuine data
Some of the options I can think of are:
1. Reduce 30days hot+warm and move to cold
2. increase disk space for hot storage

0 Karma

splunker969
Communicator

Thanks! koshyk

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...