Deployment Architecture

Help with the buckets and hot/cold data settings

gsypsomos
Engager

Need 12 months hot data, 3 months cold, nothing else

I put the following in /opt/splunk/etc/system/local/indexes.conf:
[main]
frozenTimePeriodInSecs = 39312000

That setting is supposed to remove anything over 1.25 years old in my data.

Then I restarted splunk, but the size of the indexes did not go down and I still have less than 500MB remaining in my partition, so that server is not accepting input from forwarders. The files taking up 60% of that space are in /local/splunk/hot/named_application/db* files, and did not change after restarting the server.

Shouldn't the setting added to indexes.conf have removed anything over 39312000 seconds (1.25 years) old from my indexes? I am using Splunk 6.5.2.

The documentation from Splunk is a convoluted mess. Please don't answer by saying "Read this" and pointing me to a user manual.

Thanks for your help,
George

Rob2520
Communicator

Hi George,

Have you tried this. Assuming your index is sized appropriately(maxTotalDataSizeMB).

In indexes.conf

[main]
homePath.maxDataSizeMB=31557600(1 year in seconds)
coldPath.maxDataSizeMB=7889400(3 months in seconds)
frozenTimePeriodInSecs = 39450000(15 months)

0 Karma

DavidHourani
Super Champion

homePath.maxDataSizeMB is in MB not in seconds 🙂 

Careful not to confuse both or you may end up misconfiguring your environment.

You want homePath.maxDataSizeMB + coldPath.maxDataSizeMB to be equal to maxDataSizeMB and you want to make sure that maxDataSizeMB is big enough to fit "frozenTimePeriodInSecs " worth of logs in seconds. 

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

I'm going to do it anyway... From our documentation:

I ran out of disk space so I changed the archive policy, but it's still not working
If you changed your archive policy to be more restrictive because you've run out of disk space, you may notice that events haven't started being archived according to your new policy. This is most likely because you must first free up some space so the process has room to run. Stop the indexer, clear out ~5GB of disk space, and then start the indexer again. After a while (exactly how long depends on how much data there is to process) you should see INFO entries about BucketMover in splunkd.log showing that buckets are being archived.

0 Karma

earlhelms
Path Finder

Consider using btool to verify that the setting that you changed is the one actually being used.

0 Karma

lfedak_splunk
Splunk Employee
Splunk Employee

Hey @gsypsomos, The buckets don't roll until the last record of each bucket reaches the time specified, so that might be your issue (RE: That setting is supposed to remove anything over 1.25 years old in my data.). As for eliminating the warm bucket and frozen bucket -- you can change your configuration files on those, but I believe the rule of thumb is to definitely use the warm bucket because the search will check the time range on each bucket to see if it should search for results inside of the bucket and can skip the entire bucket if it does not fit that range. This wiki has some more info: https://wiki.splunk.com/Deploy:BucketRotationAndRetention

0 Karma

gjanders
SplunkTrust
SplunkTrust

The documentation from Splunk is a
convoluted mess

If you have constructive feedback you can send feedback to the documentation team, they often reply and / or update the documentation based on feedback!
I've requested quite a few pages to be updated to make the documentation to be more clear, however some of the pages are auto-generated from the codebase so documentation updates can take a while.

0 Karma

kellewic
Path Finder

frozenTimePeriodInSecs only applies to cold data, not hot data. You'll want to check out maxHotSpanSecs, maxWarmDBCount, maxHotBuckets, maxHotIdleSecs, et al to control hot/warm buckets.

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...