Deployment Architecture

Best practices for hot/warm bucket retention?

splunker969
Communicator

The primary indexers data (Hot+ Warm) data is being full .Please help us in solving this issues . .We are trying to shrink the hot and warm are our primary indexers .
The retention period for hot +warm is 30 days.

What are best practice process? To come out of this issue .

somesoni2
Revered Legend

Data in hot/warm buckets are not managed by Splunk's data retention policy. Data is only frozen (archived or deleted based on what you've configured) from cold bucket directory. If you want to reduce the amount of data you want to retain for your index, ensure that you've configuration to have warm buckets rolled to cold buckets. See this for list of attributes that affect the data buckets:
http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Configureindexstorage#Attributes_that_affe...

Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.

0 Karma

splunker969
Communicator

Thanks @somesoni2

0 Karma

Sukisen1981
Champion

Correctly explained in detail by @somesoni2 . The data retention policy of 30 days is not a guarantee that ALL data in a 30 day period will be stored in the hot / wam buckets. What is important is to know how much data you are indexing. Things can get a bit complex here. Your organization will pay Splunk for the maximum amount of data indexed in a 24 hour period, but since you are not receiving any license violation warnings, I assume that is not a concern at this point of time.
@somesoni2 says - Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.
You need to make a decision on what size (how much GB) of your data is needed in hot and warm buckets.

0 Karma

splunker969
Communicator

Thanks @sukisen1981

0 Karma

koshyk
Super Champion

I'm not sure what you meant by Shrink? That's genuine data
Some of the options I can think of are:
1. Reduce 30days hot+warm and move to cold
2. increase disk space for hot storage

0 Karma

splunker969
Communicator

Thanks! koshyk

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...